How to access CSV data set config in my Java Request Sampler - performance
I have to read data from a csv file which contains 10000+ records.
I want to use this data in JMeter to hit a web service.
I had written my code with the hard coded value. But I want to make it dynamic.
How can I access CSV data set config in my custom Java Request Sampler...?
How can I access the variable i declared in the CSV data set config in my java request sampler..?
Here is my full code :
#Override
public SampleResult runTest(JavaSamplerContext arg0)
{
SampleResult result = new SampleResult();
boolean success = true;
byte arr[] = new byte[] {1,49,45,1,2,(byte)214,1,1,98,0,6,0,0,9,24,0,0,0,0,0,0,0,0,0,0,0,0,0,127,(byte)255,0,21,0,16,0,75,1,0,0,58,32,2,7,0,0,4,4,0,85,81,98,0,5,14,(byte)158,0,2,0,0,0,0,0,88,82,50,69,49,83,49,86,48,67,48,0,0,1,97,75,0,84,30,12,7,17,5,7,50,0,0,0,0,0,0,0,0,0,0,0,0,5,0,0,0,0,0,0,0,0,4,6,0,0,48,49,48,48,49,48,51,48,0,0,0,0,0,0,0,0,0,0,70,48,10,29,22,85,0,1,(byte)134,(byte)160,(byte)255,(byte)255,(byte)158,(byte)170,0,0,0,67,0,0,0,0,2,0,0,12,0,12,0,12,0,12,0,13,0,12,0,13,0,12,0,13,0,12,0,13,0,13,0,12,0,12,0,12,0,12,0,13,0,12,0,13,0,13,0,12,0,13,0,13,0,12,0,13,0,13,0,12,0,13,0,13,0,13,0,12,0,13,0,13,0,13,0,14,0,13,0,12,0,13,0,13,0,13,0,13,0,12,0,13,0,13,0,13,0,14,0,13,0,13,0,13,0,12,0,13,0,13,2,(byte)158,2,(byte)159,2,(byte)241,2,(byte)234,5,48,5,68,8,90,7,(byte)193,6,15,4,10,3,100,4,(byte)224,7,47,6,72,4,(byte)170,4,4,4,7,5,16,6,107,6,114,5,(byte)195,4,(byte)179,2,(byte)198,0,13,0,13,0,13,0,14,0,13,0,13,0,14,0,13,0,14,0,13,0,13,0,14,0,13,0,13,0,14,0,13,0,14,0,13,0,14,0,13,0,14,0,13,0,13,0,101,0,99,0,(byte)129,0,(byte)129,2,81,2,(byte)224,1,(byte)153,0,(byte)30,0,31,0,14,0,13,0,14,0,13,0,14,0,14,0,13,0,14,0,13,0,14,0,14,0,21,0,86,0,98,0,51,0,72,0,104,0,(byte)144,0,(byte)175,0,(byte)174,0,(byte)174,2,20,4,(byte)132,4,103,5,96,0,126,0,14,0,14,0,14,0,14,0,14,0,15,0,14,0,14,0,14,0,14,0,14,0,85,1,41,1,104,0,14,0,14,0,13,0,14,0,14,0,14,0,13,0,14,0,14,0,14,0,13,0,14,0,14,0,13,0,14,0,14,0,13,0,14,0,14,0,13,0,14,0,14,0,13,0,14,0,13,0,14,0,14,0,14,0,13,0,14,0,14,0,13,0,14,0,14,0,13,0,14,0,14,0,13,0,14,0,14,0,14,0,13,0,14,0,14,0,13,0,14,0,14,0,13,0,14,0,14,0,14,0,13,0,14,0,14,0,14,0,13,0,14,0,14,0,14,0,13,0,14,0,14,0,13,0,14,0,14,0,14,0,13,0,14,0,14,0,14,0,13,0,14,0,14,0,14,0,13,0,14,0,14,0,14,0,13,0,14,0,14,0,14,0,13,0,14,0,14,0,13,0,14,0,14,0,13,0,14,0,14,0,14,0,13,0,14,0,13,0,14,0,13,0,14,0,13,0,14,0,13,0,14,0,14,0,13,0,14,0,13,0,13,0,13,0,13,0,13,0,12,0,13,0,13,0,14,0,13,0,13,0,13,0,13,0,13,0,13,0,13,0,14,0,13,0,13,0,14,0,13,0,13,0,14,0,13,0,13,0,13,0,13,0,13,0,(byte)226,0,(byte)223,0,(byte)223,0,15,0,14,0,13,0,115,0,(byte) 223,(byte)162,40,38,85,115,101,114,78,97,109,101,61,101,82,101,103,38,85,115,101,114,80,97,115,115,119,111,114,100,61,97,98,99,49,50,51};
try
{
URL obj = new URL(POST_URL);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
con.setRequestMethod("POST");
con.setDoOutput(true);
result.sampleStart();
OutputStream os = con.getOutputStream();
os.write(arr);
os.flush();
os.close();
result.sampleEnd();
int responseCode = con.getResponseCode();
System.out.println("POST Response Code :: " + responseCode);
if (responseCode == HttpURLConnection.HTTP_OK)
{ //success
BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null)
{
response.append(inputLine);
}
in.close();
// print result
System.out.println(response.toString().getBytes());
}
else
{
System.out.println("POST request not worked");
}
}
catch(Exception E)
{
}
//
result.setSuccessful(success);
return result;
}
#Override
public Arguments getDefaultParameters()
{
Arguments dp=new Arguments();
return dp;
}
#Override
public void setupTest(JavaSamplerContext context) {}
#Override
public void teardownTest(JavaSamplerContext context) {
}
Normally you should be able to access JMeter Variables like:
String myVar = JMeterContextService.getContext().getVariables().get("your_variable_name_here");
However if you don't want to have it hard-coded you might consider moving the configuration to Java Request Sampler GUI like:
String valueFromCsv = "";
String defaultValue = "insert_jmeter_variable_here";
#Override
public Arguments getDefaultParameters() {
Arguments dp = new Arguments();
dp.addArgument("hexData", "insert_jmeter_variable_here");
return dp;
}
#Override
public void setupTest(JavaSamplerContext context) {
valueFromCsv = context.getParameter("hexData", defaultValue );
}
This way you will be able to control the parameter value directly from JMeter GUI.
References:
JMeterContextService JavaDoc
Java Request Sampler documentation
Extending JMeter
Beanshell vs JSR223 vs Java JMeter Scripting: The Performance-Off You've Been Waiting For!
A full java sampler solution with CSV data:
courtesy - https://dzone.com/articles/implement-custom-jmeter-samplers
Created a class
public class VDCSampler extends AbstractJavaSamplerClient implements Serializable {
private static final String ARG1_IDATE = "idate";
private String attrib1;
#Override
public Arguments getDefaultParameters() {
Arguments defaultParameters = new Arguments();
defaultParameters.addArgument(ARG1_IDATE, attrib1);
return defaultParameters;
}
#Override
public void setupTest(JavaSamplerContext javaSamplerContext) {
attrib1 = javaSamplerContext.getParameter(ARG1_IDATE, attrib1);
}
#Override
public SampleResult runTest(JavaSamplerContext javaSamplerContext) {
VDCFunctionalitySampling functionalityForSampling = new VDCFunctionalitySampling();
SampleResult sampleResult = new SampleResult();
sampleResult.sampleStart();
try {
String message = functionalityForSampling.testFunction(attrib1);
}
}
Related
Loading value from json upon start up application
I want to load the values from json file upon the Spring Boot Application is started. My code for the Configuration File is like the below: #Configuration #Getter public class FedexAPIConfig { private final static String JSON_FILE = "/static/config/fedex-api-credentials.json"; private final boolean IS_PRODUCTION = false; private FedexAPICred apiCredentials; public FedexAPIConfig() { try (InputStream in = getClass().getResourceAsStream(JSON_FILE); BufferedReader reader = new BufferedReader(new InputStreamReader(in, StandardCharsets.UTF_8))) { JSONObject json = new JSONObject(); // this.apiCredentials = new JSONObject(new JSONTokener(reader)); if (IS_PRODUCTION) { json = new JSONObject(new JSONTokener(reader)).getJSONObject("production"); } else { json = new JSONObject(new JSONTokener(reader)).getJSONObject("test"); } System.out.println(json.toString()); this.apiCredentials = FedexAPICred.builder() .url(json.optString("url")) .apiKey(json.optString("api_key")) .secretKey(json.optString("secret_key")) .build(); } catch (FileNotFoundException fnfe) { fnfe.printStackTrace(); } catch (IOException ioe) { ioe.printStackTrace(); } catch (Exception e) { e.printStackTrace(); } } } and with this, when the application is in progress of startup, values are successfully printed on the console.Startup console log When I tried to call this value from other ordinary class, like the below:, it brings nothing but just throws NullPointerException... What are my faults and what shall I do? public class FedexOAuthTokenManager extends OAuthToken { private static final String VALIDATE_TOKEN_URL = "/oauth/token"; private static final String GRANT_TYPE_CLIENT = "client_credentials"; private static final String GRANT_TYPE_CSP = "csp_credentials"; #Autowired private FedexAPIConfig fedexApiConfig; #Autowired private Token token; #Override public void validateToken() { // This is the part where "fedexApiConfig" is null. FedexAPICred fedexApiCred = fedexApiConfig.getApiCredentials(); Response response = null; try { RequestBody body = new FormBody.Builder() .add("grant_type", GRANT_TYPE_CLIENT) .add("client_id", fedexApiCred.getApiKey()) .add("client_secret", fedexApiCred.getSecretKey()) .build(); response = new HttpClient().post(fedexApiCred.getUrl() + VALIDATE_TOKEN_URL, body); if (response.code() == 200) { JSONObject json = new JSONObject(response.body().string()); token.setAccessToken(json.optString("access_token")); token.setTokenType(json.optString("token_type")); token.setExpiredIn(json.optInt("expires_in")); token.setExpiredDateTime(LocalDateTime.now().plusSeconds(json.optInt("expires_in"))); token.setScope(json.optString("scope")); } } catch (IOException ioe) { ioe.printStackTrace(); } } } fedexApiConfg is null even though I autowired it in prior to call. And this FedexOAuthTokenManager is called from other #Component class by new FedexOAuthTokenManager()
Did you try like below? Step 1: Create one Configuration class like below public class DemoConfig implements ApplicationListener<ApplicationPreparedEvent> { #Override public void onApplicationEvent(ApplicationPreparedEvent event) { //Load the values from the JSON file and populate the application //properties dynamically ConfigurableEnvironment environment = event.getApplicationContext().getEnvironment(); Properties props = new Properties(); props.put("spring.datasource.url", "<my value>"); //Add more properties environment.getPropertySources().addFirst(new PropertiesPropertySource("myProps", props)); } To listen to a context event, a bean should implement the ApplicationListener interface which has just one method onApplicationEvent().The ApplicationPreparedEvent is invoked very early in the lifecycle of the application Step 2: Customize in src/main/resources/META-INF/spring.factories org.springframework.context.ApplicationListener=com.example.demo.DemoConfig Step 3: #Value in spring boot is commonly used to inject the configuration values into the spring boot application. Access the properties as per your wish. #Value("${spring.datasource.url}") private String valueFromJSon; Try this sample first in your local machine and then modify your changes accordingly. Refer - https://www.baeldung.com/spring-value-annotation Refer - https://www.knowledgefactory.net/2021/02/aws-secret-manager-service-as.html
How to implement PayUmonney in Android & how to create a Hash key in local because I don't know how to create in server
public void PayuMonney(){ **JAVA this method i am using in my code but it's not done here i am use payumonney code given to Documentation https://payumobile.gitbook.io/sdk-integration/android/payucheckoutpro. i trying too much but the toast appear in "invalid Hash"** PayUPaymentParams.Builder builder = new PayUPaymentParams.Builder(); builder.setAmount(mAmount) .setIsProduction(true) .setProductInfo(mProductInfo) .setKey(mMerchantKey) .setPhone(mPhoneNumber) .setTransactionId(mTXNId) .setFirstName(mFirstName) .setEmail(mEmailId) .setSurl("https://www.payumoney.com/mobileapp/payumoney/success.php") .setFurl("https://www.payumoney.com/mobileapp/payumoney/failure.php"); **Optional can contain any additional PG params** PayUPaymentParams payUPaymentParams = builder.build(); ***here i am calling Payuminney checkout process Sample code*** PayUCheckoutPro.open( this, payUPaymentParams, new PayUCheckoutProListener() { #Override public void onPaymentSuccess(Object response) { //Cast response object to HashMap HashMap<String,Object> result = (HashMap<String, Object>) response; String payuResponse = (String)result.get(PayUCheckoutProConstants.CP_PAYU_RESPONSE); String merchantResponse = (String) result.get(PayUCheckoutProConstants.CP_MERCHANT_RESPONSE); } #Override public void onPaymentFailure(Object response) { //Cast response object to HashMap HashMap<String,Object> result = (HashMap<String, Object>) response; String payuResponse = (String)result.get(PayUCheckoutProConstants.CP_PAYU_RESPONSE); String merchantResponse = (String) result.get(PayUCheckoutProConstants.CP_MERCHANT_RESPONSE); } #Override public void onPaymentCancel(boolean isTxnInitiated) { } #Override public void onError(ErrorResponse errorResponse) { *//code give some error toast here i the onError fuction* String errorMessage = errorResponse.getErrorMessage(); Toast.makeText(FinalPlaceOrderActivity.this, errorMessage, Toast.LENGTH_SHORT).show(); } #Override public void setWebViewProperties(#Nullable WebView webView, #Nullable Object o) { *//For setting webview properties, if any. Check Customized Integration section for more details on this* } #Override public void generateHash(HashMap<String, String> valueMap, PayUHashGenerationListener hashGenerationListener) { String hashName = valueMap.get(PayUCheckoutProConstants.CP_HASH_NAME); String hashData = valueMap.get(PayUCheckoutProConstants.CP_HASH_STRING); if (!TextUtils.isEmpty(hashName) && !TextUtils.isEmpty(hashData)) { *//Do not generate a hash from local, it needs to be calculated from server-side only. Here, hashString contains hash created from your server side. //here i am call the server file where generate a hash* StringRequest OrderPlace = new StringRequest(Request.Method.POST, "url link", new Response.Listener() { #Override public void onResponse(String response) { //here i am get hash from the server System.out.println(response); String merchandHsh = response; HashMap<String, String> dataMap = new HashMap<>(); dataMap.put(hashName, merchandHsh); hashGenerationListener.onHashGenerated(dataMap); } }, new Response.ErrorListener() { #Override public void onErrorResponse(VolleyError error) { Toast.makeText(getApplicationContext(), "We can't process this time..", Toast.LENGTH_SHORT).show(); } } ) { #Nullable #Override protected Map<String, String> getParams() throws AuthFailureError { Map<String, String> map = new HashMap<String, String>(); map.put("key", mMerchantKey); map.put("texID", mTXNId); map.put("amount", mAmount); map.put("productname", mProductInfo); map.put("name", mFirstName); map.put("email", mEmailId); return map; } }; Volley.newRequestQueue(FinalPlaceOrderActivity.this).add(OrderPlace); } } } ); } but the toast appears a "Hash invalid". this method I am using in my code but it's not done here I am using the pay money code given to Documentation https://payumobile.gitbook.io/sdk-integration/android/payucheckoutpro. I trying too much but the toast appears in "invalid Hash" How to implement PayUmonney in Android & how to create a Hash key in local because I don't know how to create in server
Get ProcessGroup from processor
Is there an API to get a ProcessGroup by id from a custom processor or an ExecuteScript processor? I know that it is possible by using the REST-API, but for security reasons, I don't have the chance to use the credentials to invoke the API from a service. Regards
If you use a InvokeScriptedProcessor using groovy you can use context.procNode.processGroup from the ProcessContext If you want to extract all the parents in a breadcrum way, you can use this: import groovy.json.*; import org.apache.nifi.groups.*; class GroovyProcessor implements Processor { def REL_SUCCESS = new Relationship.Builder() .name("success") .description('FlowFiles that were successfully processed are routed here').build() def ComponentLog log #Override void initialize(ProcessorInitializationContext context) { log = context.logger } #Override Set<Relationship> getRelationships() { return [REL_SUCCESS] as Set } void executeScript(ProcessSession session, ProcessContext context) { def flowFile = session.get() if (!flowFile) { return } def breadcrumb = getBreadCrumb(context.procNode.processGroup) + '->' + context.getName() flowFile = session.putAttribute(flowFile, 'breadcrumb', breadcrumb) // transfer session.transfer(flowFile, this.REL_SUCCESS) } // Recursive funtion that gets the breadcrumb String getBreadCrumb(processGroup) { def breadCrumb = '' if(processGroup.parent != null) breadCrumb = getBreadCrumb(processGroup.parent) + '->' return breadCrumb + processGroup.name } #Override void onTrigger(ProcessContext context, ProcessSessionFactory sessionFactory) throws ProcessException { def session = sessionFactory.createSession() try { executeScript( session, context) session.commit() } catch (final Throwable t) { log.error('{} failed to process due to {}; rolling back session', [this, t] as Object[]) session.rollback(true) throw t } } #Override PropertyDescriptor getPropertyDescriptor(String name) { null } #Override List<PropertyDescriptor> getPropertyDescriptors() { return [] as List } #Override void onPropertyModified(PropertyDescriptor descriptor, String oldValue, String newValue) { } #Override Collection<ValidationResult> validate(ValidationContext context) { null } #Override String getIdentifier() { null } } processor = new GroovyProcessor()
How to make queryparams mandatory in Java Jersey REST services?
I have a REST API that accepts 3 query params. When the query is called without any one of the query parameters, the API executes and returns the result. How do we make the queryparams mandatory? How can I add validation to check if all the parameters are present? Also, please let me know the best approach.
On a very simple level you could just inject the HttpServletRequest and check yourself: #GET public Response example(#Context HttpServletRequest request, #QueryParam("name") String name) { if (null == request.getParameter("name")) { ResponseBuilder builder = Response.status(404); return builder.build(); } // Do something with name } Or you can implement something more elaborate using AOP. Here's a blog post about further options.
jersey doesn't give a mandatory parameter checking functionality out of the box. however you can do something like implementing your own annotation to achieve it. Below is the annotation code: #Target(value = ElementType.METHOD) #Retention(value = RetentionPolicy.RUNTIME) public #interface Required { String[] value(); } You also need a filter, below is the code: public class RequiredParamResourceFilterFactory implements ResourceFilterFactory { #Context private transient HttpServletRequest servletRequest; private class RequiredParamFilter implements ResourceFilter, ContainerRequestFilter { private final String[] requiredParams; protected List<String> parametersValueMissing; private RequiredParamFilter(String[] requiredParams) { this.requiredParams = requiredParams; } #Override public ContainerRequest filter(ContainerRequest containerRequest) { boolean missingMandatoryParameter = false; List<String> missingParameters = new ArrayList<String>(); List<String> requiredParametersValueMissing = new ArrayList<String>(); List<String> URLParameters = getURLParameters(containerRequest.getQueryParameters()); List<String> methodRequiredParameters = Arrays.asList(requiredParams); if (methodRequiredParameters != null) { for (String methodRequiredParam : methodRequiredParameters) { if (URLParameters == null) { missingMandatoryParameter = true; //we will check this flag before returning result set to caller missingParameters.add(methodRequiredParam); } else if (!URLParameters.contains(methodRequiredParam)) { missingMandatoryParameter = true; //we will check this flag before returning result set to caller missingParameters.add(methodRequiredParam); //Add to required parameters value missing List, only if the parameter is mandatory and value is not provided // in the URL } else if (parametersValueMissing.contains(methodRequiredParam)) { requiredParametersValueMissing.add(methodRequiredParam); } } if (missingMandatoryParameter && requiredParametersValueMissing.size() > 0) { throw new YourCustomException("Missing Parameters = " + StringHelper.ArrayToString(missingParameters) + "\nParameter value missing for " + StringHelper.ArrayToString(requiredParametersValueMissing)); } else if (missingMandatoryParameter) { throw new YourCustomException("Missing Parameters = " + StringHelper.ArrayToString(missingParameters), MisbarErrorCode.VALIDATION_WRONG_INPUT_ERROR, "Customers"); } else if (requiredParametersValueMissing != null && requiredParametersValueMissing.size() > 0) { throw new YourCustomException("Parameter value missing for " + StringHelper.ArrayToString(requiredParametersValueMissing)); } } return containerRequest; } #Override public ContainerRequestFilter getRequestFilter() { return this; } #Override public ContainerResponseFilter getResponseFilter() { return null; } /** * To fetch the parameters sent to webservice call, these will be used to find if required parameter * are present or not * * #param queryParams the queryparams sent * #return all the parameters sent in URL */ private List<String> getURLParameters(MultivaluedMap<String,String> queryParams) { parametersValueMissing = new ArrayList<String>(); List<String> arr = new ArrayList<String>(); for(String key:queryParams.keySet()) { arr.add(key); if(queryParams.get(key)==null) parametersValueMissing.add(key); } if(!arr.isEmpty()) return arr; return null; } } #Override public List<ResourceFilter> create(AbstractMethod am) { Required required = am.getAnnotation(Required.class); if(required!=null) { return Collections.<ResourceFilter>singletonList(new RequiredParamFilter(required.value())); } return null; } } Below sample shows how to use this annotation, so in below webservice; file_id and count are mandatory parameters: #GET #Produces(MediaType.APPLICATION_JSON+";charset=utf-8") #Cacheable(isCacheable = true) #Path("posts/clusters") #Required({"file_id","count"}) #Timed public Response getClusters( #QueryParam("file_id") Integer fileId, #QueryParam("count") Integer count, #DefaultValue("-1")#QueryParam("start_time") Long startTime){ ; } If mandatory parameters are not provided in webservice call, you receive an error like below, mentioning the parameter names that are missing: { message: "Missing Parameters = file_id, count", errorCode: "600" } Hope this solves your problem.
Custom WritableCompare displays object reference as output
I am new to Hadoop and Java, and I feel there is something obvious I am just missing. I am using Hadoop 1.0.3 if that means anything. My goal for using hadoop is to take a bunch of files and parse them one file at a time (as opposed to line by line). Each file will produce multiple key-values, but context to the other lines is important. The key and value are multi-value/composite, so I have implemented WritableCompare for the key and Writable for the value. Because the processing of each file take a bit of CPU, I want to save the output of the mapper, then run multiple reducers later on. For the composite keys, I followed [http://stackoverflow.com/questions/12427090/hadoop-composite-key][1] The problem is, the output is just Java object references as opposed to the composite key and value. Example: LinkKeyWritable#bd2f9730 LinkValueWritable#8752408c I am not sure if the problem is related to not reducing the data at all or Here is my main class: public static void main(String[] args) throws Exception { JobConf conf = new JobConf(Parser.class); conf.setJobName("raw_parser"); conf.setOutputKeyClass(LinkKeyWritable.class); conf.setOutputValueClass(LinkValueWritable.class); conf.setMapperClass(RawMap.class); conf.setNumMapTasks(0); conf.setInputFormat(PerFileInputFormat.class); conf.setOutputFormat(TextOutputFormat.class); PerFileInputFormat.setInputPaths(conf, new Path(args[0])); FileOutputFormat.setOutputPath(conf, new Path(args[1])); JobClient.runJob(conf); } And my Mapper class: public class RawMap extends MapReduceBase implements Mapper { public void map(NullWritable key, Text value, OutputCollector<LinkKeyWritable, LinkValueWritable> output, Reporter reporter) throws IOException { String json = value.toString(); SerpyReader reader = new SerpyReader(json); GoogleParser parser = new GoogleParser(reader); for (String page : reader.getPages()) { String content = reader.readPageContent(page); parser.addPage(content); } for (Link link : parser.getLinks()) { LinkKeyWritable linkKey = new LinkKeyWritable(link); LinkValueWritable linkValue = new LinkValueWritable(link); output.collect(linkKey, linkValue); } } } Link is basically a struct of various information that get's split between LinkKeyWritable and LinkValueWritable LinkKeyWritable: public class LinkKeyWritable implements WritableComparable<LinkKeyWritable>{ protected Link link; public LinkKeyWritable() { super(); link = new Link(); } public LinkKeyWritable(Link link) { super(); this.link = link; } #Override public void readFields(DataInput in) throws IOException { link.batchDay = in.readLong(); link.source = in.readUTF(); link.domain = in.readUTF(); link.path = in.readUTF(); } #Override public void write(DataOutput out) throws IOException { out.writeLong(link.batchDay); out.writeUTF(link.source); out.writeUTF(link.domain); out.writeUTF(link.path); } #Override public int compareTo(LinkKeyWritable o) { return ComparisonChain.start(). compare(link.batchDay, o.link.batchDay). compare(link.domain, o.link.domain). compare(link.path, o.link.path). result(); } #Override public int hashCode() { return Objects.hashCode(link.batchDay, link.source, link.domain, link.path); } #Override public boolean equals(final Object obj){ if(obj instanceof LinkKeyWritable) { final LinkKeyWritable o = (LinkKeyWritable)obj; return Objects.equal(link.batchDay, o.link.batchDay) && Objects.equal(link.source, o.link.source) && Objects.equal(link.domain, o.link.domain) && Objects.equal(link.path, o.link.path); } return false; } } LinkValueWritable: public class LinkValueWritable implements Writable{ protected Link link; public LinkValueWritable() { link = new Link(); } public LinkValueWritable(Link link) { this.link = new Link(); this.link.type = link.type; this.link.description = link.description; } #Override public void readFields(DataInput in) throws IOException { link.type = in.readUTF(); link.description = in.readUTF(); } #Override public void write(DataOutput out) throws IOException { out.writeUTF(link.type); out.writeUTF(link.description); } #Override public int hashCode() { return Objects.hashCode(link.type, link.description); } #Override public boolean equals(final Object obj){ if(obj instanceof LinkKeyWritable) { final LinkKeyWritable o = (LinkKeyWritable)obj; return Objects.equal(link.type, o.link.type) && Objects.equal(link.description, o.link.description); } return false; } }
I think the answer is in the implementation of the TextOutputFormat. Specifically, the LineRecordWriter's writeObject method: /** * Write the object to the byte stream, handling Text as a special * case. * #param o the object to print * #throws IOException if the write throws, we pass it on */ private void writeObject(Object o) throws IOException { if (o instanceof Text) { Text to = (Text) o; out.write(to.getBytes(), 0, to.getLength()); } else { out.write(o.toString().getBytes(utf8)); } } As you can see, if your key or value is not a Text object, it calls the toString method on it and writes that out. Since you've left toString unimplemented in your key and value, it's using the Object class's implementation, which is writing out the reference. I'd say that you should try writing an appropriate toString function or using a different OutputFormat.
It looks like you have a list of objects just like you wanted. You need to implement toString() on your writable if you want a human-readable version printed out instead of an ugly java reference.