Everytime I run the application on server and enter the url I don't get any response. It says the following (image):
Java code is as follows:
package application;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import javax.ws.rs.GET;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.Path;
import javax.ws.rs.core.Application;
import javax.ws.rs.ApplicationPath;
#Path("demo")
public class FRITZ {
#GET
#Produces(MediaType.TEXT_PLAIN)
public String calculateCellComplexities(String spreadsheetPath) throws Exception {
System.out.print("Entered Cell Complexity......\n");
//open spreadsheet
Spreadsheet spreadsheet;
try {
spreadsheet = POIReader.readFromFile(spreadsheetPath);
} catch (Exception e) {
e.printStackTrace();
return null;
}
//analyze metrics
AnalyzeEngine analyzer = new AnalyzeEngine();
boolean success;
try {
success = analyzer.initializeEvaluationMode(EvaluationMode.SMELLS_COMPLETE);
analyzer.analyze(spreadsheet);
} catch (Exception e) {
e.printStackTrace();
return null;
}
if(!success) {
System.err.println("Could not analyze spreadsheet: abort!");
return null;
}
//prepare metrics for analysis
List<Metric> analysisMetrics = new ArrayList<Metric>();
analysisMetrics.add(MetricFactory.getMetric(LongCalculationChainSmell.class));
analysisMetrics.add(MetricFactory.getMetric(PatternFinderColumnNoBorder.class));
analysisMetrics.add(MetricFactory.getMetric(PatternFinderRowNoBorder.class));
analysisMetrics.add(MetricFactory.getMetric(MultipleReferences.class));
analysisMetrics.add(MetricFactory.getMetric(MultipleOperations.class));
//initialize max metric value data
Double[] maxValues = new Double[analysisMetrics.size()];
for(int i = 0; i < maxValues.length; i++) {
maxValues[i] = 0.0d;
}
//determine max metric values in spreadsheet
for(Worksheet worksheet : spreadsheet.getWorksheets()) {
for(Cell cell : worksheet.getCells()) {
for(int index = 0; index < analysisMetrics.size(); index++) {
Metric metric = analysisMetrics.get(index);
if(metric.domain == Domain.INTEGER) {
Integer metricValue = (Integer) metric.getMetricValueForCell(cell);
if(metricValue != null)
{
maxValues[index] = Math.max(maxValues[index], metricValue);
}
} if(metric.domain == Domain.LONG) {
Long metricValue = (Long) metric.getMetricValueForCell(cell);
if(metricValue != null)
{
maxValues[index] = Math.max(maxValues[index], metricValue);
}
} if (metric.domain == Domain.REAL) {
Double metricValue = (Double) metric.getMetricValueForCell(cell);
if(metricValue != null)
maxValues[index] = Math.max(maxValues[index], metricValue);
}
}
}
}
// for(int index = 0; index < analysisMetrics.size(); index++) {
// Metric metric = analysisMetrics.get(index);
// if(metric.domain == Domain.INTEGER || metric.domain == Domain.REAL) {
// System.out.println(index + ": " + maxValues[index]);
// }
// }
//calculate complexity estimate
Map<String,Double> probabilityMap = new HashMap<String, Double>();
for(Worksheet worksheet : spreadsheet.getWorksheets()) {
for(Cell cell : worksheet.getCells()) {
double complexity = 0.0d;
if(cell.isFormulaCell()) {
int numAppliedMetrics = 0;
for(int index = 0; index < analysisMetrics.size(); index++) {
Metric metric = analysisMetrics.get(index);
switch(metric.domain) {
case BOOLEAN:
Boolean booleanMetricValue = (Boolean) metric.getMetricValueForCell(cell);
if(booleanMetricValue != null) {
complexity += (booleanMetricValue ? 1.0f : 0.0f);
numAppliedMetrics++;
}
break;
case INTEGER:
Integer metricValue = (Integer) metric.getMetricValueForCell(cell);
if(metricValue != null) {
complexity += metricValue / maxValues[index];
numAppliedMetrics++;
}
break;
case REAL:
Double doubleMetricValue = (Double) metric.getMetricValueForCell(cell);
if(doubleMetricValue != null) {
complexity += doubleMetricValue / maxValues[index];
numAppliedMetrics++;
}
break;
case STRING:
default:
break;
}
}
if(numAppliedMetrics > 0)
complexity /= numAppliedMetrics;
}
probabilityMap.put(cell.getPosition().toA1DebugString(), complexity);
}
}
//return probabilityMap;
String a = "hello from server";
return a;
}
}
Please ignore main logic of function, I am just testing web api with simple returned string, once it runs successuflly I will return the json.
Following is my web.xml file:
<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://xmlns.jcp.org/xml/ns/javaee" xsi:schemaLocation="http://xmlns.jcp.org/xml/ns/javaee http://xmlns.jcp.org/xml/ns/javaee/web-app_3_1.xsd" id="WebApp_ID" version="3.1">
<display-name>FRITZ-WEBAPI</display-name>
<servlet>
<servlet-name>FRITZ JAVA API</servlet-name>
<servlet-class>org.glassfish.jersey.servlet.ServletContainer.class</servlet-class>
<init-param>
<param-name>jersey.config.server.provider.packages</param-name>
<param-value>application</param-value>
</init-param>
<load-on-startup>1</load-on-startup>
</servlet>
<servlet-mapping>
<servlet-name>FRITZ JAVA API</servlet-name>
<url-pattern>/rest/*</url-pattern>
</servlet-mapping>
</web-app>
Related
I have the below requirement.
I need to perform the sum of each field across multiple records of the same entity However while performing the sum, I also need to check the type and cast them accrodingly. For eg, For whole number cast to Int, For Decimal cast to decimal. Also some of the values are aliased value too. I am looking for a generic function which I can call for both alias fields and direct fields and it will return me the value based on the type
Background on the code written below -
Attribute List is the list of all attributes that belong to the
entity.
Format in which the field values are stored in AttributeList-
AttributeList = { "price ", "quantity", "contact.revenue", "opportunity.sales"}
price, quantity - fields of main entity on which we are querying
contact.revenue, opportunity.sales - fields of the aliased entities,
entity name is appended to understand which entity's field it is
Below is the code which i have tried so far -
I only have decimal and whole number fields in my attributeList.
private void calculate(List<string> attributeList,List<Entity> mainEntityList,Guid targetId,Guid oppId,Guid contactId)
{
var mainentity = new mainEntity();
mainentity.Id = targetId;
var opportunity = new Opportunity();
opportunity.Id = oppId;
var contact = new Contact();
contact.Id = contactId;
foreach (var attribute in attributeList)
{
var fieldSum = new decimal(0);
int intFieldSum = 0;
bool attributeFound = false;
foreach (var entity in mainEntityList)
{
if (entity.Contains(attribute))
{
var type = entity[attribute].GetType().Name;
attributeFound = true;
switch (type)
{
case "AliasedValue":
var aliasedFieldValue = entity.GetAttributeValue<AliasedValue>(attribute);
if (aliasedFieldValue.Value.GetType().Name == "Decimal")
{
decimalFieldSum += (decimal)aliasedFieldValue.Value;
}
else
{
intFieldSum += (int)aliasedFieldValue.Value;
}
break;
case "Decimal":
decimalFieldSum += entity.GetAttributeValue<decimal>(attribute);
break;
case "Int32":
intFieldSum += entity.GetAttributeValue<int>(attribute);
break;
default:
break;
}
}
}
if (attributeFound)
{
if (attribute.Contains("opportunity"))
{
opportunity[attribute] = decimalFieldSum != 0 ? decimalFieldSum : intFieldSum;
}
else if (attribute.Contains("contact"))
{
contact[attribute] = decimalFieldSum != 0 ? decimalFieldSum : intFieldSum;
}
else
{
mainentity[attribute] = decimalFieldSum != 0 ? decimalFieldSum : intFieldSum;
}
}
}
service.update(opportunity);
service.update(contact);
service.update(mainentity);
}
Any help would be appreciated.
Just a little bit edited your code.
...
var fieldSum = new decimal(0);
foreach (var entity in mainEntityList)
{
fieldSum += GetAttrValue(entity, attribute);
}
...
You can use this function to calculate fieldSum variable which is of decimal type.
private decimal GetAttrValue(Entity entity, string attribute)
{
var attrValue = new decimal(0);
if (!entity.Contains(attribute) || entity.Attributes[attribute] == null)
{
return attrValue;
}
var type = entity.Attributes[attribute].GetType().Name;
switch (type)
{
case "AliasedValue":
var aliasedFieldValue = entity.GetAttributeValue<AliasedValue>(attribute);
attrValue = type == "Decimal" ? (decimal)aliasedFieldValue.Value : (int)aliasedFieldValue.Value;
break;
case "Decimal":
attrValue = entity.GetAttributeValue<decimal>(attribute);
break;
case "Int32":
attrValue = entity.GetAttributeValue<int>(attribute);
break;
default:
break;
}
return attrValue;
}
On the other hand if you just need a generic function which will return decimal or int value for an attribute you can use this
private T GetAttrValue<T>(Entity entity, string attribute)
{
if (!entity.Contains(attribute) || entity.Attributes[attribute] == null)
{
return default(T);
}
T result;
var type = entity.Attributes[attribute].GetType().Name;
if (type == "AliasedValue")
{
var aliasedFieldValue = entity.GetAttributeValue<AliasedValue>(attribute);
result = (T)aliasedFieldValue.Value;
}
else
{
result = entity.GetAttributeValue<T>(attribute);
}
return result;
}
--Update--
So, here is the whole code if I understand you requirements right.
First of all add this class.
public class AttributeInfo
{
public string Name { get; set; }
public Type Type { get; set; }
public decimal DecimalSum { get; set; } = new decimal(0);
public int IntSum { get; set; } = 0;
}
And add this function
private void SetValue(Entity entity, AttributeInfo attributeInfo)
{
if (entity.Contains(attributeInfo.Name))
{
switch (attributeInfo.Type.Name)
{
case "Decimal":
entity[attributeInfo.Name] = attributeInfo.DecimalSum;
break;
case "Int32":
entity[attributeInfo.Name] = attributeInfo.IntSum;
break;
default:
break;
}
}
}
Then this is you Calculate function
private void Calculate(List<string> attributeList, List<Entity> mainEntityList, Guid targetId, Guid oppId, Guid contactId)
{
var mainentity = new mainEntity();
mainentity.Id = targetId;
var opportunity = new Opportunity();
opportunity.Id = oppId;
var contact = new Contact();
contact.Id = contactId;
var attributesInfo = new List<AttributeInfo>();
foreach (var attribute in attributeList)
{
var attributeInfo = new AttributeInfo
{
Name = attribute
};
foreach (var entity in mainEntityList)
{
if (entity.Contains(attribute))
{
attributeInfo.Type = entity[attribute].GetType();
switch (attributeInfo.Type.Name)
{
case "AliasedValue":
var aliasedFieldValue = entity.GetAttributeValue<AliasedValue>(attribute);
if (aliasedFieldValue.Value.GetType().Name == "Decimal")
{
attributeInfo.DecimalSum += (decimal)aliasedFieldValue.Value;
}
else
{
attributeInfo.IntSum += (int)aliasedFieldValue.Value;
}
break;
case "Decimal":
attributeInfo.DecimalSum += entity.GetAttributeValue<decimal>(attribute);
break;
case "Int32":
attributeInfo.IntSum += entity.GetAttributeValue<int>(attribute);
break;
default:
break;
}
}
}
attributesInfo.Add(attributeInfo);
}
foreach (var attributeInfo in attributesInfo)
{
if (attributeInfo.Type != null)
{
SetValue(mainentity, attributeInfo);
SetValue(opportunity, attributeInfo);
SetValue(contact, attributeInfo);
}
}
service.update(mainentity);
service.update(opportunity);
service.update(contact);
}
I should say that the structure of the calculate function still seems weird for me. However, here I tried to keep the main structure.
I am building a game in Spring Boot on a server and classic Javascript on a backend.
Right now I have this:
...
#Autowired
private SimpMessagingTemplate template;
...
#Scheduled(fixedRate = 1000 / Constants.FPS)
public void renderClients() {
for(Game g : games) {
template.convertAndSend("/game/render/" + g.getId(), g);
}
}
...
Basically I have a multiple Games running and I send each with it's id to the client.
However the data I am sending (or the most of the data) is static (not changing)...
What if I want not to send the whole data but only parts which have changed.
Btw the response JSON looks like this:
{"id":"862b1dd8-48d5-4562-802a-7d669a5a5ed5","players":[{"id":"da8dcbec-7028-4a39-9547-a4e2dc321c3c","name":"John Doe","position":{"x":100.0,"y":100.0},"rotation":0.0,"hero":{"maxHealth":1300.0,"movementSpeed":4.5,"attackDamage":32.75,"width":68,"height":71,"heroName":"drowRanger","radius":34.0},"stats":{"kills":0,"lastHits":0},"lastClick":null}],"duration":380107.12}
and the only thing that is changing is duration and sometimes the x and y when the player moves...
Is it even possible?
Could I write some middleware that will do that at the time the objects are converted to JSON?
Maintain a data structure stores your changed value, and attach it to your Game Object.
When the time to send ,convert the map to a json ,and clear it.
Using this way may use more memory than before , but won't cost much time.
I DID IT!!
In my GameController I do:
#Scheduled(fixedRate = 1000 / Constants.FPS)
public void renderClients() throws Exception {
for(Game g : games) {
template.convertAndSend("/game/render/" + g.getId(), g.formatToSend());
}
}
Notice the g.formatToSend() method
here is how a Game class looks like:
public class Game {
private BandWidthOptimizer optimizer = new BandWidthOptimizer();
...
...
public String formatToSend() throws Exception {
return optimizer.optimize(this);
}
}
And Here Comes THE BandWidthOptimizer:
package com.iddqd.doto.optimization;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.sun.tools.classfile.Opcode;
import org.json.simple.JSONArray;
import org.json.simple.JSONObject;
import org.json.simple.parser.JSONParser;
import java.util.Iterator;
import java.util.Set;
import java.util.function.BiConsumer;
public class BandWidthOptimizer {
import com.fasterxml.jackson.databind.ObjectMapper;
import org.json.simple.JSONArray;
import org.json.simple.JSONObject;
import org.json.simple.parser.JSONParser;
private String[] preserveKeys;
public BandWidthOptimizer() {
this.preserveKeys = new String[0];
}
public BandWidthOptimizer(String[] preserveKeys) {
this.preserveKeys = preserveKeys;
}
public String optimize(Object obj) throws Exception {
String json = mapper.writeValueAsString(obj);
Object nobj = parser.parse(json);
Object oobj = parser.parse(lastJSON);
JSONObject newJsonObj = (JSONObject)nobj;
JSONObject oldJsonObj = (JSONObject)oobj;
JSONObject res = getJSONObjectDiff(newJsonObj, oldJsonObj);
lastJSON = json;
return res.toJSONString();
}
private JSONObject getJSONObjectDiff(JSONObject obj1, JSONObject obj2) {
JSONObject res = new JSONObject();
Set set = obj1.keySet();
for (Object key : set) {
// If doesn't exist put it in the diff
if (!obj2.containsKey(key)) {
res.put(key, obj1.get(key));
} else {
// Get the values from both objects
Object val1 = obj1.get(key);
Object val2 = obj2.get(key);
// If their instances are of the same type
if(val1 == null) {
continue;
}
if(val2 == null) {
res.put(key, val1);
continue;
}
if (val1.getClass().equals(val2.getClass())) {
// If they are JSONObject
if (val1 instanceof JSONObject) {
// Recursively parse JSONObject with all of it's properties
JSONObject nested = getJSONObjectDiff((JSONObject) obj1.get(key), (JSONObject) obj2.get(key));
// If it contains any keys
if(nested.keySet().size() > 0) {
// Store the diff into final diff
res.put(key, nested);
}
// If they are JSONArrays
} else if (val1 instanceof JSONArray) {
// If val1 contains some values (is not empty)
if(((JSONArray) val1).size() > 0) {
// Get their diff
JSONArray arr = getJSONArrayDiff((JSONArray) val1, (JSONArray) val2);
// If array is not empty
if (arr.size() > 0) {
// put it into the diff
res.put(key, arr);
}
}
// If they are just a pure values
} else {
// Compare them - If they're not equal
if(!val1.equals(val2)) {
// put the val1 into diff
res.put(key, val1);
}
}
} else {
res.put(key, val1);
}
}
}
return res;
}
private JSONArray getJSONArrayDiff(JSONArray arr1, JSONArray arr2) {
JSONArray res = new JSONArray();
// For every element
for(int i = 0; i < arr1.size(); i++) {
Object val1 = arr1.get(i);
// If i is out of arr2 bounds
if(i > arr2.size()) {
// put the arr1 item into the diff
res.add(val1);
}
Object val2 = arr2.get(i);
if(val1 == null) {
continue;
}
if(val2 == null) {
res.add(val1);
continue;
}
// If their types are equal
if(val1.getClass().equals(val2.getClass())) {
// If they are JSONObjects
if(val1 instanceof JSONObject) {
// Get their diff
JSONObject obj = getJSONObjectDiff((JSONObject) val1, (JSONObject) val2);
// If it contains any keys
if(obj.keySet().size() > 0) {
// Store the diff into final diff
res.add(obj);
}
// If they are JSONArrays
} else if (val1 instanceof JSONArray) {
// Get their diff
JSONArray arr = getJSONArrayDiff((JSONArray) val1, (JSONArray) val2);
// If array is not empty
if(arr.size() > 0) {
// put it into the diff
res.add(arr);
}
// If they are just a pure values
} else {
// Compare them - If they're not equal
if(val1 != val2) {
// add the val1 into diff
res.add(val1);
}
}
} else {
res.add(val1);
}
}
return res;
}
}
This is it, now if nothing moves on the map the result JSON looks like this:
{"duration":282964.56}
because only the duration changes
But when my Player moves on the map see what happens:
{"duration":386676.06,"players":[{"position":{"x":556.5914801003707,"y":153.55964799554002}}]}
TODO
I have to implement a preserveKeys functionallity because I always want to send some keys like id and so on...
I created Web API project for a maze game service. It uses WebApi.Hal for returning hal+jsonand hal+xml media types. It works for the most part – except for the following issues.
Though page is 2 in the request, it returns all the cells for all pages. How to correct it?
In hal+json, there are “links” to all cells in the self itself, in addition to embedded cells. In hal+xml, there is only embedded cells. Why is the difference? Which is correct?
There is a link added as rel="page" href="~/api/cells/page/{page}" . Why is the rel coming as “page” though I have written following code
`public static Link GetCells { get { return new Link("cell","~/api/cells/page/{page}"); } }
There is a link added with searchTerm .How to utilize it in the client? I could not find any useful example.
All the cells are coming as embedded. Is it the right way for HAL collection? Or should it be links onl?
Request: http://localhost:51910/api/cells/page/2
application/hal+xml
<?xml version="1.0" encoding="utf-8"?>
<resource rel="" href="~/api/cells/page/2">
<link rel="prev" href="~/api/cells/page/1" />
<link rel="next" href="~/api/cells/page/3" />
<link rel="page" href="~/api/cells/page/{page}" />
<link rel="page" href="~/api/cells{?searchTerm,page}" />
<TotalResults>6</TotalResults>
<TotalPages>3</TotalPages>
<Page>2</Page>
<resource rel="self" href="~/api/cells/00">
<link rel="left" href="~/api/cells/10" />
<link rel="left" href="~/api/cells/01" />
<CellID>00</CellID>
<IsExit>False</IsExit>
</resource>
<resource rel="self" href="~/api/cells/10">
<link rel="down" href="~/api/cells/00" />
<link rel="down" href="~/api/cells/20" />
<CellID>10</CellID>
<IsExit>False</IsExit>
</resource>
<resource rel="self" href="~/api/cells/20">
<link rel="left" href="~/api/cells/10" />
<link rel="right" href="~/api/cells/21" />
<CellID>20</CellID>
<IsExit>False</IsExit>
</resource>
<resource rel="self" href="~/api/cells/01">
<link rel="down" href="~/api/cells/00" />
<link rel="left" href="~/api/cells/11" />
<CellID>01</CellID>
<IsExit>False</IsExit>
</resource>
<resource rel="self" href="~/api/cells/11">
<link rel="left" href="~/api/cells/01" />
<link rel="right" href="~/api/cells/21" />
<CellID>11</CellID>
<IsExit>False</IsExit>
</resource>
<resource rel="self" href="~/api/cells/21">
<link rel="down" href="~/api/cells/20" />
<link rel="left" href="~/api/cells/11" />
<CellID>21</CellID>
<IsExit>False</IsExit>
</resource>
</resource>
application/hal+json
Controller
public class CellsController : ApiController
{
//Get all cells in the maze
[HttpGet]
[Route("api/cells")]
public CellListRep Get()
{
Maze m = new Maze();
int page = 1;
int totalPages = 1;
List<CellRepresentation> cellRepresentations = GetAllCellRepresentation(m);
int totalCells = cellRepresentations.Count;
return new CellListRep(cellRepresentations, totalCells, totalPages, page, LinkTemplates.CellLinks.GetCells);
}
//Get paged list
[HttpGet]
[Route("api/cells/page/{page}")]
public CellListRep Get(int page)
{
Maze m = new Maze();
List<CellRepresentation> cellRepresentations = GetAllCellRepresentation(m);
int totalCells = cellRepresentations.Count;
int totalPages = cellRepresentations.Count / 2;
return new CellListRep(cellRepresentations, totalCells, totalPages, page, LinkTemplates.CellLinks.GetCells);
}
//Get specific cell
[HttpGet]
[Route("api/cells/{id}")]
public CellRepresentation Get(string id)
{
Maze m = new Maze();
Cell c = m.GetSpecificCell(id);
List<Cell> possibleLists = m.GetPossibleRelatedCells(c);
CellRepresentation beerRep = new CellRepresentation(c, possibleLists);
return beerRep;
}
private List<CellRepresentation> GetAllCellRepresentation(Maze m)
{
List<CellRepresentation> cellRepresentations = new List<CellRepresentation>();
List<Cell> cells = m.GetAllCells();
foreach (Cell c in cells)
{
List<Cell> possibleLists = m.GetPossibleRelatedCells(c);
CellRepresentation beerRep = new CellRepresentation(c, possibleLists);
cellRepresentations.Add(beerRep);
}
return cellRepresentations;
}
}
Representations and Links
public static class LinkTemplates
{
public static class CellLinks
{
public static Link CellEntry { get { return new Link("self", "~/api/cells/{id}"); } }
public static Link UpLink { get { return new Link("up", "~/api/cells/{id}"); } }
public static Link RightLink { get { return new Link("right", "~/api/cells/{id}"); } }
public static Link DownLink { get { return new Link("down", "~/api/cells/{id}"); } }
public static Link LeftLink { get { return new Link("left", "~/api/cells/{id}"); } }
public static Link GetCells { get { return new Link("cell", "~/api/cells/page/{page}"); } }
public static Link SearchCells { get { return new Link("page", "~/api/cells{?searchTerm,page}"); } }
}
}
public class CellRepresentation : WebApi.Hal.Representation
{
Cell theCell;
List<Cell> possibleMoveCells;
public String CellID
{
get
{
return theCell.CellID ;
}
}
public bool IsExit
{
get
{
return theCell.IsExtCell;
}
}
public CellRepresentation(Cell c, List<Cell> possibleMoveCells )
{
theCell = c;
//_cellIDVal = c.CellID;
this.possibleMoveCells = possibleMoveCells;
}
public override string Rel
{
get { return LinkTemplates.CellLinks.CellEntry.Rel; }
set { }
}
public override string Href
{
get { return LinkTemplates.CellLinks.CellEntry.CreateLink(new { id = theCell.CellID }).Href; }
set { }
}
protected override void CreateHypermedia()
{
foreach (Cell relatedCell in possibleMoveCells)
{
if (relatedCell.RelativeName == "Up")
{
Links.Add(LinkTemplates.CellLinks.UpLink.CreateLink(new { id = relatedCell.CellID }));
}
if (relatedCell.RelativeName == "Right")
{
Links.Add(LinkTemplates.CellLinks.RightLink.CreateLink(new { id = relatedCell.CellID }));
}
if (relatedCell.RelativeName == "Down")
{
Links.Add(LinkTemplates.CellLinks.DownLink.CreateLink(new { id = relatedCell.CellID }));
}
if (relatedCell.RelativeName == "Left")
{
Links.Add(LinkTemplates.CellLinks.LeftLink.CreateLink(new { id = relatedCell.CellID }));
}
}
}
}
public class CellListRep : PagedRepresentationList<CellRepresentation>
{
public CellListRep(IList<CellRepresentation> beers, int totalResults, int totalPages, int page, Link uriTemplate) :
base(beers, totalResults, totalPages, page, uriTemplate, null)
{ }
public CellListRep(IList<CellRepresentation> beers, int totalResults, int totalPages, int page, Link uriTemplate,
object uriTemplateSubstitutionParams) :
base(beers, totalResults, totalPages, page, uriTemplate, uriTemplateSubstitutionParams)
{ }
protected override void CreateHypermedia()
{
base.CreateHypermedia();
var search = LinkTemplates.CellLinks.SearchCells;
if (Links.Count(l => l.Rel == search.Rel && l.Href == search.Href) == 0)
{
Links.Add(LinkTemplates.CellLinks.SearchCells);
}
}
}
public abstract class PagedRepresentationList<TRepresentation> : SimpleListRepresentation<TRepresentation> where TRepresentation : Representation
{
readonly Link uriTemplate;
protected PagedRepresentationList(IList<TRepresentation> res, int totalResults, int totalPages, int page, Link uriTemplate, object uriTemplateSubstitutionParams)
: base(res)
{
this.uriTemplate = uriTemplate;
TotalResults = totalResults;
TotalPages = totalPages;
Page = page;
UriTemplateSubstitutionParams = uriTemplateSubstitutionParams;
}
public int TotalResults { get; set; }
public int TotalPages { get; set; }
public int Page { get; set; }
protected object UriTemplateSubstitutionParams;
protected override void CreateHypermedia()
{
var prms = new List<object> { new { page = Page } };
if (UriTemplateSubstitutionParams != null)
prms.Add(UriTemplateSubstitutionParams);
Href = Href ?? uriTemplate.CreateLink(prms.ToArray()).Href;
Links.Add(new Link { Href = Href, Rel = "self" });
if (Page > 1)
{
var item = UriTemplateSubstitutionParams == null
? uriTemplate.CreateLink("prev", new { page = Page - 1 })
: uriTemplate.CreateLink("prev", UriTemplateSubstitutionParams, new { page = Page - 1 }); // page overrides UriTemplateSubstitutionParams
Links.Add(item);
}
if (Page < TotalPages)
{
var link = UriTemplateSubstitutionParams == null // kbr
? uriTemplate.CreateLink("next", new { page = Page + 1 })
: uriTemplate.CreateLink("next", UriTemplateSubstitutionParams, new { page = Page + 1 }); // page overrides UriTemplateSubstitutionParams
Links.Add(link);
}
Links.Add(new Link("page", uriTemplate.Href));
}
}
Business Classes
public class Cell
{
public int XVal { get; set; }
public int YVal { get; set; }
public bool TopIsWall { get; set; }
public bool RightIsWall { get; set; }
public bool BottomIsWall { get; set; }
public bool LeftIsWall { get; set; }
public bool IsStartCell { get; set; }
public bool IsExtCell { get; set; }
public string RelativeName { get; set; } //Top, Right, Etc.
public string CellID
{
get
{
string characterID = XVal.ToString() + YVal.ToString();
return characterID; //Example 10
}
}
}
public class Maze
{
List<Cell> cells;
public Maze()
{
cells = CreateCells();
}
public Cell GetFirtCell()
{
Cell firstCell = null;
foreach (Cell c in cells)
{
if(c.IsStartCell )
{
firstCell = c;
break;
}
}
return firstCell;
}
public Cell GetSpecificCell(string cellID)
{
Cell theCell = null;
foreach (Cell c in cells)
{
if (c.CellID == cellID)
{
theCell = c;
break;
}
}
return theCell;
}
public List<Cell> GetAllCells()
{
return cells;
}
public List<Cell> GetPossibleRelatedCells(Cell inputCell)
{
List<Cell> possibleCells = new List<Cell>();
foreach (Cell c in cells)
{
if (c.YVal == 2 && c.XVal == 3)
{
int test = 0;
}
if (c.XVal == inputCell.XVal-1 && c.RightIsWall == false && c.YVal== inputCell.YVal )
{
//Go left from the input cell
c.RelativeName = "Left";
possibleCells.Add(c);
}
else if (c.XVal == inputCell.XVal + 1 && c.LeftIsWall == false && c.YVal == inputCell.YVal )
{
//Go right from the input cell
c.RelativeName = "Right";
possibleCells.Add(c);
}
else if (c.YVal == inputCell.YVal - 1 && c.TopIsWall == false && c.XVal == inputCell.XVal )
{
//Go down from the input cell
c.RelativeName = "Down";
possibleCells.Add(c);
}
else if (c.YVal == inputCell.YVal + 1 && c.BottomIsWall == false && c.XVal == inputCell.XVal)
{
//Go up from the input cell
c.RelativeName = "Up";
possibleCells.Add(c);
}
}
return possibleCells;
}
public List<Cell> CreateCells()
{
List<Cell> cells = new List<Cell>();
Cell cell1 = new Cell
{
XVal = 0,YVal = 0,TopIsWall = false,RightIsWall = false,BottomIsWall = true,LeftIsWall = true,
RelativeName="Self"
};
Cell cell2 = new Cell
{
XVal = 1,YVal = 0,TopIsWall = true,RightIsWall = false,BottomIsWall = false,LeftIsWall = false,
IsStartCell = true, //--Start
RelativeName="Self"
};
Cell cell3 = new Cell
{
XVal = 2,YVal = 0,TopIsWall = false,RightIsWall = false,BottomIsWall = true,LeftIsWall = false,
RelativeName="Self"
};
Cell cell5 = new Cell
{
XVal = 0,YVal = 1,TopIsWall = true,RightIsWall = false,BottomIsWall = false,LeftIsWall = true,
RelativeName = "Self"
};
Cell cell7 = new Cell
{
XVal = 1,YVal = 1,TopIsWall = true,RightIsWall = false,BottomIsWall = true,LeftIsWall = false,
RelativeName = "Self"
};
Cell cell8 = new Cell
{
XVal = 2,YVal = 1,TopIsWall = false,RightIsWall = true,BottomIsWall = false,LeftIsWall = false,
RelativeName = "Self"
};
cells.Add(cell1);
cells.Add(cell2);
cells.Add(cell3);
cells.Add(cell5);
cells.Add(cell7);
cells.Add(cell8);
return cells;
}
}
I need your help to know how to use an alias (stored tuple) on my Pig udf function, i exmplain:
my_file.csv
101,message here
102,message here
103,message here
...
My script PIG:
X = load'mydata.csv' using PigStorage(',') as (myVar:chararray);
A = load'my_file.csv' using PigStorage(',') as (key:chararray,value:chararray);
B = GROUP par ALL;
C = foreach B {
D = ORDER par BY key;
GENERATE BagToTuple(D);
};
the result of the C is something like (101,message here, 102, message here, 103, message here...)
Now what i need is to pass this result in my udf function like :
Z = foreach X generate MYUDF(myVar, C);
the alias "C" is the tuple key,value,key,value...
MYUDF :
import java.io.IOException;
import java.util.regex.Pattern;
import org.apache.pig.EvalFunc;
import org.apache.pig.data.Tuple;
import org.apache.pig.PigWarning;
import org.apache.pig.data.DataType;
import org.apache.pig.impl.util.WrappedIOException;
import org.apache.pig.impl.logicalLayer.schema.Schema;
public class ReDecode extends EvalFunc<String> {
int numParams = -1;
Pattern mPattern = null;
#Override
public Schema outputSchema(Schema input) {
try {
return new Schema(new Schema.FieldSchema(getSchemaName(this
.getClass().getName().toLowerCase(), input),
DataType.CHARARRAY));
} catch (Exception e) {
return null;
}
}
#Override
public String exec(Tuple tuple) throws IOException {
if (numParams==-1) // Not initialized
{
numParams = tuple.size();
if (numParams <= 2) {
String msg = "Decode: Atleast an expression and default string is required.";
throw new IOException(msg);
}
if (tuple.size()%2!=0) {
String msg = "ItssPigUDFs.ReDecode : Some parameters are unmatched.";
throw new IOException(msg);
}
}
if (tuple.get(0)==null)
return null;
try {
for (int count = 1; count < numParams - 1; count += 2)
{
mPattern=Pattern.compile((String)tuple.get(count));
if (mPattern.matcher((String)tuple.get(0)).matches())
{
return (String)tuple.get(count+1);
}
}
} catch (ClassCastException e) {
warn("ItssPigUDFs.ReDecode : Data type error", PigWarning.UDF_WARNING_1);
return null;
} catch (NullPointerException e) {
String msg = "ItssPigUDFs.ReDecode : Encounter null in the input";
throw new IOException(msg);
}
return (String)tuple.get(tuple.size()-1);
}
Thank you for your help
I don't think numParams is needed; the number of params that you get to the UDF will be input.size().
Therefore, if you call MYUDF(myVar, C), then you should be able to get those values in Java like String myVar = (String) input.get(0) and Tuple param2 = input.get(1).
Is it possible to change the record delimiter from newline to some other string so as to read a file with newlines into a single tuple in pig.
Yes.
A = LOAD '...' USING PigStorage(',') AS (...); //comma is the delimeter for fields
SET textinputformat.record.delimiter '<delimeter>'; // record delimeter, by default it is `\n`. You can change to any delimeter.
As mentioned here
You can use PigStorage
A = LOAD '/some/path/COMMA-DELIM-PREFIX*' USING PigStorage(',') AS (f1:chararray, ...);
B = LOAD '/some/path/SEMICOLON-DELIM-PREFIX*' USING PigStorage('\t') AS (f1:chararray, ...);
You can even try writing load/store UDF.
There is java code example for both load and store.
Load Functions : LoadFunc abstract class has the main methods for loading data and for most use cases it would suffice to extend it. You can read more here
Example
The loader implementation in the example is a loader for text data
with line delimiter as '\n' and '\t' as default field delimiter (which
can be overridden by passing a different field delimiter in the
constructor) - this is similar to current PigStorage loader in Pig.
The implementation uses an existing Hadoop supported Inputformat -
TextInputFormat - as the underlying InputFormat.
public class SimpleTextLoader extends LoadFunc {
protected RecordReader in = null;
private byte fieldDel = '\t';
private ArrayList<Object> mProtoTuple = null;
private TupleFactory mTupleFactory = TupleFactory.getInstance();
private static final int BUFFER_SIZE = 1024;
public SimpleTextLoader() {
}
/**
* Constructs a Pig loader that uses specified character as a field delimiter.
*
* #param delimiter
* the single byte character that is used to separate fields.
* ("\t" is the default.)
*/
public SimpleTextLoader(String delimiter) {
this();
if (delimiter.length() == 1) {
this.fieldDel = (byte)delimiter.charAt(0);
} else if (delimiter.length() > 1 & & delimiter.charAt(0) == '\\') {
switch (delimiter.charAt(1)) {
case 't':
this.fieldDel = (byte)'\t';
break;
case 'x':
fieldDel =
Integer.valueOf(delimiter.substring(2), 16).byteValue();
break;
case 'u':
this.fieldDel =
Integer.valueOf(delimiter.substring(2)).byteValue();
break;
default:
throw new RuntimeException("Unknown delimiter " + delimiter);
}
} else {
throw new RuntimeException("PigStorage delimeter must be a single character");
}
}
#Override
public Tuple getNext() throws IOException {
try {
boolean notDone = in.nextKeyValue();
if (!notDone) {
return null;
}
Text value = (Text) in.getCurrentValue();
byte[] buf = value.getBytes();
int len = value.getLength();
int start = 0;
for (int i = 0; i < len; i++) {
if (buf[i] == fieldDel) {
readField(buf, start, i);
start = i + 1;
}
}
// pick up the last field
readField(buf, start, len);
Tuple t = mTupleFactory.newTupleNoCopy(mProtoTuple);
mProtoTuple = null;
return t;
} catch (InterruptedException e) {
int errCode = 6018;
String errMsg = "Error while reading input";
throw new ExecException(errMsg, errCode,
PigException.REMOTE_ENVIRONMENT, e);
}
}
private void readField(byte[] buf, int start, int end) {
if (mProtoTuple == null) {
mProtoTuple = new ArrayList<Object>();
}
if (start == end) {
// NULL value
mProtoTuple.add(null);
} else {
mProtoTuple.add(new DataByteArray(buf, start, end));
}
}
#Override
public InputFormat getInputFormat() {
return new TextInputFormat();
}
#Override
public void prepareToRead(RecordReader reader, PigSplit split) {
in = reader;
}
#Override
public void setLocation(String location, Job job)
throws IOException {
FileInputFormat.setInputPaths(job, location);
}
}
Store Functions : StoreFunc abstract class has the main methods for storing data and for most use cases it should suffice to extend it
Example
The storer implementation in the example is a storer for text data
with line delimiter as '\n' and '\t' as default field delimiter (which
can be overridden by passing a different field delimiter in the
constructor) - this is similar to current PigStorage storer in Pig.
The implementation uses an existing Hadoop supported OutputFormat -
TextOutputFormat as the underlying OutputFormat.
public class SimpleTextStorer extends StoreFunc {
protected RecordWriter writer = null;
private byte fieldDel = '\t';
private static final int BUFFER_SIZE = 1024;
private static final String UTF8 = "UTF-8";
public PigStorage() {
}
public PigStorage(String delimiter) {
this();
if (delimiter.length() == 1) {
this.fieldDel = (byte)delimiter.charAt(0);
} else if (delimiter.length() > 1delimiter.charAt(0) == '\\') {
switch (delimiter.charAt(1)) {
case 't':
this.fieldDel = (byte)'\t';
break;
case 'x':
fieldDel =
Integer.valueOf(delimiter.substring(2), 16).byteValue();
break;
case 'u':
this.fieldDel =
Integer.valueOf(delimiter.substring(2)).byteValue();
break;
default:
throw new RuntimeException("Unknown delimiter " + delimiter);
}
} else {
throw new RuntimeException("PigStorage delimeter must be a single character");
}
}
ByteArrayOutputStream mOut = new ByteArrayOutputStream(BUFFER_SIZE);
#Override
public void putNext(Tuple f) throws IOException {
int sz = f.size();
for (int i = 0; i < sz; i++) {
Object field;
try {
field = f.get(i);
} catch (ExecException ee) {
throw ee;
}
putField(field);
if (i != sz - 1) {
mOut.write(fieldDel);
}
}
Text text = new Text(mOut.toByteArray());
try {
writer.write(null, text);
mOut.reset();
} catch (InterruptedException e) {
throw new IOException(e);
}
}
#SuppressWarnings("unchecked")
private void putField(Object field) throws IOException {
//string constants for each delimiter
String tupleBeginDelim = "(";
String tupleEndDelim = ")";
String bagBeginDelim = "{";
String bagEndDelim = "}";
String mapBeginDelim = "[";
String mapEndDelim = "]";
String fieldDelim = ",";
String mapKeyValueDelim = "#";
switch (DataType.findType(field)) {
case DataType.NULL:
break; // just leave it empty
case DataType.BOOLEAN:
mOut.write(((Boolean)field).toString().getBytes());
break;
case DataType.INTEGER:
mOut.write(((Integer)field).toString().getBytes());
break;
case DataType.LONG:
mOut.write(((Long)field).toString().getBytes());
break;
case DataType.FLOAT:
mOut.write(((Float)field).toString().getBytes());
break;
case DataType.DOUBLE:
mOut.write(((Double)field).toString().getBytes());
break;
case DataType.BYTEARRAY: {
byte[] b = ((DataByteArray)field).get();
mOut.write(b, 0, b.length);
break;
}
case DataType.CHARARRAY:
// oddly enough, writeBytes writes a string
mOut.write(((String)field).getBytes(UTF8));
break;
case DataType.MAP:
boolean mapHasNext = false;
Map<String, Object> m = (Map<String, Object>)field;
mOut.write(mapBeginDelim.getBytes(UTF8));
for(Map.Entry<String, Object> e: m.entrySet()) {
if(mapHasNext) {
mOut.write(fieldDelim.getBytes(UTF8));
} else {
mapHasNext = true;
}
putField(e.getKey());
mOut.write(mapKeyValueDelim.getBytes(UTF8));
putField(e.getValue());
}
mOut.write(mapEndDelim.getBytes(UTF8));
break;
case DataType.TUPLE:
boolean tupleHasNext = false;
Tuple t = (Tuple)field;
mOut.write(tupleBeginDelim.getBytes(UTF8));
for(int i = 0; i < t.size(); ++i) {
if(tupleHasNext) {
mOut.write(fieldDelim.getBytes(UTF8));
} else {
tupleHasNext = true;
}
try {
putField(t.get(i));
} catch (ExecException ee) {
throw ee;
}
}
mOut.write(tupleEndDelim.getBytes(UTF8));
break;
case DataType.BAG:
boolean bagHasNext = false;
mOut.write(bagBeginDelim.getBytes(UTF8));
Iterator<Tuple> tupleIter = ((DataBag)field).iterator();
while(tupleIter.hasNext()) {
if(bagHasNext) {
mOut.write(fieldDelim.getBytes(UTF8));
} else {
bagHasNext = true;
}
putField((Object)tupleIter.next());
}
mOut.write(bagEndDelim.getBytes(UTF8));
break;
default: {
int errCode = 2108;
String msg = "Could not determine data type of field: " + field;
throw new ExecException(msg, errCode, PigException.BUG);
}
}
}
#Override
public OutputFormat getOutputFormat() {
return new TextOutputFormat<WritableComparable, Text>();
}
#Override
public void prepareToWrite(RecordWriter writer) {
this.writer = writer;
}
#Override
public void setStoreLocation(String location, Job job) throws IOException {
job.getConfiguration().set("mapred.textoutputformat.separator", "");
FileOutputFormat.setOutputPath(job, new Path(location));
if (location.endsWith(".bz2")) {
FileOutputFormat.setCompressOutput(job, true);
FileOutputFormat.setOutputCompressorClass(job, BZip2Codec.class);
} else if (location.endsWith(".gz")) {
FileOutputFormat.setCompressOutput(job, true);
FileOutputFormat.setOutputCompressorClass(job, GzipCodec.class);
}
}
}