I have been writing protobuf records to our s3 buckets. And I want to use flink dataset api to read from it. So I implemented a custom FileInputFormat to achieve this. The code is as below.
public class ProtobufInputFormat extends FileInputFormat<StandardLog.Pageview> {
public ProtobufInputFormat() {
}
private transient boolean reachedEnd = false;
#Override
public boolean reachedEnd() throws IOException {
return reachedEnd;
}
#Override
public StandardLog.Pageview nextRecord(StandardLog.Pageview reuse) throws IOException {
StandardLog.Pageview pageview = StandardLog.Pageview.parseDelimitedFrom(stream);
if (pageview == null) {
reachedEnd = true;
}
return pageview;
}
#Override
public boolean supportsMultiPaths() {
return true;
}
}
public class BatchReadJob {
public static void main(String... args) throws Exception {
String readPath1 = args[0];
ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
ProtobufInputFormat inputFormat = new ProtobufInputFormat();
inputFormat.setNestedFileEnumeration(true);
inputFormat.setFilePaths(readPath1);
DataSet<StandardLog.Pageview> dataSource = env.createInput(inputFormat);
dataSource.map(new MapFunction<StandardLog.Pageview, String>() {
#Override
public String map(StandardLog.Pageview value) throws Exception {
return value.getId();
}
}).writeAsText("s3://xxx", FileSystem.WriteMode.OVERWRITE);
env.execute();
}
}
The problem is that flink always assign one filesplit to one parallelism slot. In other word, it always process the same number of file split as the number of the parallelism.
I want to know what's the correct way of implementing custom FileInputFormat.
Thanks.
I believe the behavior you're seeing is because ExecutionJobVertex calls the FileInputFormat. createInputSplits() method with a minNumSplits parameter equal to the vertex (data source) parallelism. So if you want a different behavior, then you'd have to override the createInputSplits method.
Though you didn't say what behavior you actually wanted. If, for example, you just want one split per file, then you could override the testForUnsplittable() method in your subclass of FileInputFormat to always return true; it should also set the (protected) unsplittable boolean to true.
I'm using Resources.System.GetString(Resource.String.no_of_bags) inside my RecyclerView. This class doesn't contain reference for context/Activity. I'm getting runtime exception.
Like #Digitalsa1nt has said, you need pass the Context to your Adapter.
Here is the official demo about the usage of RecyclerView, but it doesn't refer to the Context when create an instance of PhotoAlbumAdapter.
You can do like this:
1) Add Context in the PhotoAlbumAdapter class
public class PhotoAlbumAdapter : RecyclerView.Adapter
{
Context mContext;
PhotoAlbum mPhotoAlbum;
public PhotoAlbumAdapter(Context context, PhotoAlbum photoAlbum)
{
this.mContext = context;
this.mPhotoAlbum = photoAlbum;
}
...
}
2) Pass this (this is the instance of MainActivity) to PhotoAlbumAdapter
mAdapter = new PhotoAlbumAdapter(this,mphotoAlbum);
3) Use mContext in OnBindViewHolder method
public override void OnBindViewHolder(RecyclerView.ViewHolder holder, int position)
{
mContext.Resources.GetString(Resource.String.no_of_bags);
}
I have a simple scenario in which am trying to verify some behavior when a method is called (i.e. that a certain method was called with given parameter, a function pointer in this scenario). Below are my classes:
#SpringBootApplication
public class Application {
public static void main(String[] args) {
ConfigurableApplicationContext context = SpringApplication.run(Application.class, args);
AppBootStrapper bootStrapper = context.getBean(AppBootStrapper.class);
bootStrapper.start();
}
}
#Component
public class AppBootStrapper {
private NetworkScanner networkScanner;
private PacketConsumer packetConsumer;
public AppBootStrapper(NetworkScanner networkScanner, PacketConsumer packetConsumer) {
this.networkScanner = networkScanner;
this.packetConsumer = packetConsumer;
}
public void start() {
networkScanner.addConsumer(packetConsumer::consumePacket);
networkScanner.startScan();
}
}
#Component
public class NetworkScanner {
private List<Consumer<String>> consumers = new ArrayList<>();
public void startScan(){
Executors.newSingleThreadExecutor().submit(() -> {
while(true) {
// do some scanning and get/parse packets
consumers.forEach(consumer -> consumer.accept("Package Data"));
}
});
}
public void addConsumer(Consumer<String> consumer) {
this.consumers.add(consumer);
}
}
#Component
public class PacketConsumer {
public void consumePacket(String packet) {
System.out.println("Packet received: " + packet);
}
}
#RunWith(JUnit4.class)
public class AppBootStrapperTest {
#Test
public void start() throws Exception {
NetworkScanner networkScanner = mock(NetworkScanner.class);
PacketConsumer packetConsumer = mock(PacketConsumer.class);
AppBootStrapper appBootStrapper = new AppBootStrapper(networkScanner, packetConsumer);
appBootStrapper.start();
verify(networkScanner).addConsumer(packetConsumer::consumePacket);
verify(networkScanner, times(1)).startScan();
}
}
I want to verify that bootStrapper did in fact do proper setup by registering the packet consumer(there might be other consumers registered later on, but this one is mandatory) and then called startScan. I get the following error message when I execute the test case:
Argument(s) are different! Wanted:
networkScanner bean.addConsumer(
com.spring.starter.AppBootStrapperTest$$Lambda$8/438123546#282308c3
);
-> at com.spring.starter.AppBootStrapperTest.start(AppBootStrapperTest.java:24)
Actual invocation has different arguments:
networkScanner bean.addConsumer(
com.spring.starter.AppBootStrapper$$Lambda$7/920446957#5dda14d0
);
-> at com.spring.starter.AppBootStrapper.start(AppBootStrapper.java:12)
From the exception, clearly the function pointers aren't the same.
Am I approaching this the right way? Is there something basic I am missing? I played around and had a consumer injected into PacketConsumer just to see if it made a different and that was OK, but I know that's certainly not the right way to go.
Any help, perspectives on this would be greatly appreciated.
Java doesn't have any concept of "function pointers"; when you see:
networkScanner.addConsumer(packetConsumer::consumePacket);
What Java actually compiles is (the equivalent of):
networkScanner.addConsumer(new Consumer<String>() {
#Override void accept(String packet) {
packetConsumer.consumePacket(packet);
}
});
This anonymous inner class happens to be called AppBootStrapper$$Lambda$7. Because it doesn't (and shouldn't) define an equals method, it will never be equal to the anonymous inner class that the compiler generates in your test, which happens to be called AppBootStrapperTest$$Lambda$8. This is regardless of the fact that the method bodies are the same, and are built in the same way from the same method reference.
If you generate the Consumer explicitly in your test and save it as a static final Consumer<String> field, then you can pass that reference in the test and compare it; at that point, reference equality should hold. This should work with a lambda expression or method reference just fine.
A more apt test would probably verify(packetConsumer, atLeastOnce()).consumePacket(...), as the contents of the lambda are an implementation detail and you're really more concerned about how your component collaborates with other components. The abstraction here should be at the consumePacket level, not at the addConsumer level.
See the comments and answer on this SO question.
I am using jmeter with Java Request samplers. These call java classes I have written which returns a SampleResult object which contains the timing metrics for the use case. SampleResult is a tree and can have child SampleResult objects (SampleResult.addSubResult method). I cant seem to find a good way in jmeter to track the sub results so I can only easily get the results for the parent SampleResult.
Is there a listener in jmeter that allows me to see statistics / graphs for sub results (for instance see the average time across all sub results with the same name).
I have just succeeded in doing this, and wanted to share it. If you follow the instructions I provide here, it will work for you as well. I did this for the summary table listener. And, I did it on Windows. And, I used Eclipse
Steps:
Go to JMeter's web site and download the source code. You can find that here, for version 3.0.
http://jmeter.apache.org/download_jmeter.cgi
One there, I clicked the option to download the Zip file for the Source.
Then, on that same page, download the binary for version 3.0, if you have not already done so. Then, extract that zip file onto your hard drive.
Once you've extracted the zip file to your hard drive, grab the file "SummaryReport.java". It can be found here: "\apache-jmeter-3.0\src\components\org\apache\jmeter\visualizers\SummaryReport.java"
Create a new class in Eclipse, then Copy/Paste all of that code into your new class. Then, rename your class from what it is, "SummaryReport" to a different name. And everywhere in the code, replace "SummaryReport" with the new name of your class.
I am using Java 8. So, there is one line of code that won't compile for me. It's the line below.
private final Map tableRows = new ConcurrentHashMap<>();
You need to remove the <> on that line, as Java 1.8 doesn't support it. Then, it will compile
There was one more line that gave a compile error. It was the one below.
CSVSaveService.saveCSVStats(StatGraphVisualizer.getAllTableData(model, FORMATS),writer,`
saveHeaders.isSelected() ? StatGraphVisualizer.getLabels(COLUMNS) : null);
Firstly, it wasn't finding the source for class StatGraphVisualizer. So, I imported it, as below.
import org.apache.jmeter.visualizers.StatGraphVisualizer;
Secondly, it wasn't finding the method "getLabels" in "StatGraphVisualizer.getLabels." So, here is what this line of code looked like after I fixed it. It is seen below.
CSVSaveService.saveCSVStats(StatGraphVisualizer.getAllTableData(model, FORMATS),writer);
That compiles. That method doesn't need the second argument.
Now, everything should compile.
Find this method below. This is where you will begin adding your customizations.
#Override
public void add(final SampleResult res) {
You need to create an array of all of your sub results, as I did, as seen below. The line in Bold is the new code. (All new code is seen in Bold).
public void add(final SampleResult res) {
final String sampleLabel = res.getSampleLabel(); // useGroupName.isSelected());
**final SampleResult[] theSubResults = res.getSubResults();**
Then, create a String for each label for your sub results objects, as seen below.
**final String writesampleLabel = theSubResults[0].getSampleLabel(); // (useGroupName.isSelected());
final String readsampleLabel = theSubResults[1].getSampleLabel(); // (useGroupName.isSelected());**
Next, go to the method below.
JMeterUtils.runSafe(false, new Runnable() {
#Override
public void run() {
The new code added is below, in Bold.
JMeterUtils.runSafe(false, new Runnable() {
#Override
public void run() {
Calculator row = null;
**Calculator row1 = null;
Calculator row2 = null;**
synchronized (lock) {
row = tableRows.get(sampleLabel);
**row1 = tableRows.get(writesampleLabel);
row2 = tableRows.get(readsampleLabel);**
if (row == null) {
row = new Calculator(sampleLabel);
tableRows.put(row.getLabel(), row);
model.insertRow(row, model.getRowCount() - 1);
}
**if (row1 == null) {
row1 = new Calculator(writesampleLabel);
tableRows.put(row1.getLabel(), row1);
model.insertRow(row1, model.getRowCount() - 1);
}
if (row2 == null) {
row2 = new Calculator(readsampleLabel);
tableRows.put(row2.getLabel(), row2);
model.insertRow(row2, model.getRowCount() - 1);
}**
} // close lock
/*
* Synch is needed because multiple threads can update the counts.
*/
synchronized(row) {
row.addSample(res);
}
**synchronized(row1) {
row1.addSample(theSubResults[0]);
}**
**synchronized(row2) {
row2.addSample(theSubResults[1]);
}**
That is all that needs to be customized.
In Eclipse, export your new class into a Jar file. Then place it inside of the lib/ext folder of your binary of Jmeter that you extracted, from Step 1 above.
Start up Jmeter, as you normally would.
In your Java sampler, add a new Listener. You will now see two "Summary Table" listeners. One of these will be the new one that you have just created. Once you have brought that new one into your Java Sampler, rename it to something unique. Then run your test and look at your new "Summary Table" listener. You will see summary results/stats for all of your sample results.
My next step is to perform these same steps for all of the other Listeners that I would like to customize.
I hope that this post helps.
Here is some of my plugin code which you can use as a starting point in writing your own plugin. I cant really post everything as there are really dozens of classes. Few things to know are:
my plugin like all visualizer plugins extends the jmeter class
AbstractVisualizer
you need the following jars in eclipse to complile:
jfxrt.jar,ApacheJMeter_core.jar
you need java 1.8 for javafx (the jar file comes in the sdk)
if you compile a plugin you need to put that in jmeter/lib/ext.
You also need to put the jars from bullet 2 in jmeter/lib
there is a method called "add(SampleResult)" in my class. This
will get called by the jmeter framework every time a java sample
completes and will pass the SampleResult as a parameter. Assuming you
have your own Java Sample classes that extend
AbstractJavaSamplerClient your class will have a method called
runTest which returns a sampleresult. That same return object will be
passed into your plugins add method.
my plugin puts all the sample results into a buffer and only
updates the screen every 5 results.
Here is the code:
import java.awt.BorderLayout;
import java.util.ArrayList;
import java.util.List;
import javafx.application.Platform;
import javafx.embed.swing.JFXPanel;
import javax.swing.border.Border;
import javax.swing.border.EmptyBorder;
import org.apache.jmeter.samplers.SampleResult;
import org.apache.jmeter.testelement.TestStateListener;
import org.apache.jmeter.visualizers.gui.AbstractVisualizer;
public class FxVisualizer extends AbstractVisualizer implements TestStateListener {
int currentId = 0;
/**
*
*/
private static final long serialVersionUID = 1L;
private static final int BUFFER_SIZE = 5;
#Override
public String getName()
{
return super.getName();//"George's sub result viewer.";
}
#Override
public String getStaticLabel()
{
return "Georges FX Visualizer";
}
#Override
public String getComment()
{
return "George wrote this plugin. There are many plugins like it but this one is mine.";
}
static Long initCount = new Long(0);
public FxVisualizer()
{
init();
}
private void init()
{
//LoggingUtil.debug("in FxVisualizer init()");
try
{
FxTestListener.setListener(this);
this.setLayout(new BorderLayout());
Border margin = new EmptyBorder(10, 10, 5, 10);
this.setBorder(margin);
//this.add(makeTitlePanel(), BorderLayout.NORTH);
final JFXPanel fxPanel = new JFXPanel();
add(fxPanel);
//fxPanel.setScene(getScene());
Platform.runLater(new Runnable() {
#Override
public void run() {
initFX(fxPanel);
}
});
}
catch(Exception e)
{
e.printStackTrace();
}
}
static FxVisualizerScene fxScene;
private static void initFX(JFXPanel fxPanel) {
// This method is invoked on the JavaFX thread
fxScene = new FxVisualizerScene();
fxPanel.setScene(fxScene.getScene());
}
final List <Event> bufferedEvents = new ArrayList<Event>();
#Override
public void add(SampleResult result)
{
final List <Event> events = ...;//here you need to take the result.getSubResults() parameter and get all the children events.
final List<Event> eventsToAdd = new ArrayList<Event>();
synchronized(bufferedEvents)
{
for (Event evt : events)
{
bufferedEvents.add(evt);
}
if (bufferedEvents.size() >= BUFFER_SIZE)
{
eventsToAdd.addAll(bufferedEvents);
bufferedEvents.clear();
}
}
if (eventsToAdd.size() > 0)
{
Platform.runLater(new Runnable() {
#Override
public void run() {
updatePanel(eventsToAdd);
}
});
}
}
public void updatePanel(List <Event> events )
{
for (Event evt: events)
{
fxScene.addEvent(evt);
}
}
#Override
public void clearData()
{
synchronized(bufferedEvents)
{
Platform.runLater(new Runnable() {
#Override
public void run() {
bufferedEvents.clear();
fxScene.clearData();
}
});
}
}
#Override
public String getLabelResource() {
return "Georges Java Sub FX Sample Listener";
}
Boolean isRunning = false;
#Override
public void testEnded()
{
final List<Event> eventsToAdd = new ArrayList<Event>();
synchronized(bufferedEvents)
{
eventsToAdd.addAll(bufferedEvents);
bufferedEvents.clear();
}
if (eventsToAdd.size() > 0)
{
Platform.runLater(new Runnable() {
#Override
public void run() {
updatePanel(eventsToAdd);
fxScene.testStopped();
}
});
}
}
Long testCount = new Long(0);
#Override
public void testStarted() {
synchronized(bufferedEvents)
{
Platform.runLater(new Runnable() {
#Override
public void run() {
updatePanel(bufferedEvents);
bufferedEvents.clear();
fxScene.testStarted();
}
});
}
}
#Override
public void testEnded(String arg0)
{
//LoggingUtil.debug("testEnded 2:" + arg0);
testEnded();
}
int registeredCount = 0;
#Override
public void testStarted(String arg0) {
//LoggingUtil.debug("testStarted 2:" + arg0);
testStarted();
}
}
OK so I just decided to write my own jmeter plugin and it is dead simple. Ill share the code for posterity when it is complete. Just write a class that extends AbstractVisualizer, compile it into a jar, then throw it into the jmeter lib/ext directory. That plugin will show up in the listeners section of jmeter when you go to add visualizers.
I made a method addquestion that calls addobject() and savechanges()
the Code inside the method is following:
public class AdminRepository
{
Model1Container db = new Model1Container();
CreateViewModel rep = new CreateViewModel();
public void Question addquestion() // <-- I dont know how the method should be
{
Question q = new Question();
q.QuestionText = rep.QuestionText;
db.Question.AddObject(q);
db.SaveChanges();
}
I know that the code is right but I dont know how to define the method.
Thanks in advance!
Best Regards!
You are specifying both a return type of void and Question
Since you aren't actually returning anything inside your method, the correct return type is void
public void addquestion()
{
//Your code here
}
public void AddQuestion()
{
....
}