I have an interesting issue occurring frequently in my Spring web app. Basically, a user logs into the site successfully, navigates around to protected pages for a while and then for some reason, the server starts to return 403 responses. The user calls into tech support and tech support asks them to refresh their browser and try again. Magically, everything works fine the second time around after the refresh. No one on my team is able to reproduce the issue in either our QA environment nor on Production. Does anyone have any ideas what may cause this?
UPDATE - these are logs from the apache access log file
These calls are what prompt the call to tech support (notice the 403s)
"GET /server-webapp/api/getCartContents? HTTP/1.1" 200 1273
"GET /js/config.js HTTP/1.1" 200 809
"GET /server-webapp/api/getCartContents? HTTP/1.1" 200 1273
"GET /server-webapp/api/getUserInfo? HTTP/1.1" 200 201
"GET /server-webapp/api/getPendingSalesOrder? HTTP/1.1" 200 183
"POST /server-webapp/api/getShoppingCartErrors HTTP/1.1" 200 40
"GET /server-webapp/generated/CountriesAndStates.json? HTTP/1.1" 200 3319
"GET /server-webapp/api/getAddresses? HTTP/1.1" 403 390
"POST /server-webapp/api/createPendingSalesOrder HTTP/1.1" 403 390
"GET /server-webapp/api/getAddresses?" 403 390
"POST /server-webapp/api/createPendingSalesOrder HTTP/1.1" 403 390
"GET /server-webapp/api/getAddresses? HTTP/1.1" 403 390
"POST /server-webapp/api/createPendingSalesOrder HTTP/1.1" 403 390
The user is asked to refresh their browser and those same calls are now returning 200s...
"GET /server-webapp/api/getCartContents? HTTP/1.1" 200 1273
"GET /server-webapp/api/getCartContents? HTTP/1.1" 200 1273
"GET /server-webapp/api/getUserInfo? HTTP/1.1" 200 261
"POST /server-webapp/api/getShoppingCartErrors HTTP/1.1" 200 40
"GET /server-webapp/api/getPendingSalesOrder? HTTP/1.1" 200 183
"GET /server-webapp/generated/CountriesAndStates.json? HTTP/1.1" 200 3319
"GET /server-webapp/api/getAddresses? HTTP/1.1" 200 50
"POST /server-webapp/api/createPendingSalesOrder HTTP/1.1" 200 184
"POST /server-webapp/api/updatePendingSalesOrderLines HTTP/1.1" 200 42
"GET /server-webapp/api/getPendingSalesOrder? HTTP/1.1" 200 206
The issue ended up not being with Tomcat or Apache and was, in fact, a nasty little bug in the authentication logic. In short, the user was being logged in but with no permissions but only if they took a very specific route to becoming logged in. Thanks to those that have taken a look and if there is way for me to delete the ticket then let me know since it turned out to be coding logic and not anything to do with Tomcat or Apache.
Related
when I go to my appharbor site I get a 502 Bad Gateway
in the logs I see
appharbor nginx xx.xx.xxx.xxx - "POST / HTTP/1.1" 502 150 "-" "Logplex/unknown" "source=nginx measure#http.response_time=0.000s measure#http.bytes.sent=300"
it was a problem on appharbors side
I was looking my apache logs and I find out this.
Log Line: 192.168.1.2 - - [30/Nov/2016:15:46:52 +0100] "GET http://www.Mywebsite.... HTTP/1.1" 200 5539 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
This happen a few times with that IP, but i cannot understand how is bingbot executing from that IP.
Thanks in advance
I have some strange lags issue with a sinatra thin app when the trafic get a little high (30-40 users).
It's a small game using long-polling so http IOs can be high compared to the number of users.
CPU load stay low and there's a lot of free memory.
Here are some typical logs lines when lags happen :
1 - [17/Jul/2015:16:50:17 -0400] "POST /play?next=word HTTP/1.1" 200 1 0.0018
2 - [17/Jul/2015:16:50:17 -0400] "GET /update?_=1437166100579 HTTP/1.1" 200 304 15.0046
3 - [17/Jul/2015:16:50:17 -0400] "GET /update?_=1437166102348 HTTP/1.1" 200 286 15.0045
4 - [17/Jul/2015:16:50:17 -0400] "POST /accept_replay? HTTP/1.1" 200 - 0.0021
5 - [17/Jul/2015:16:50:18 -0400] "GET /core HTTP/1.1" 200 3719 0.0015
6 - [17/Jul/2015:16:50:18 -0400] "GET /join HTTP/1.1" 302 - 0.0640
7 - [17/Jul/2015:16:50:18 -0400] "GET /core HTTP/1.1" 200 3719 0.0024
8 - [17/Jul/2015:16:50:19 -0400] "POST /play?next=word HTTP/1.1" 200 1 0.0034
9 - [17/Jul/2015:16:50:19 -0400] "GET /update?_=1437166215907 HTTP/1.1" 200 248 10.0018
10- [17/Jul/2015:16:50:19 -0400] "GET /update?_=1437166222579 HTTP/1.1" 200 252 11.0029
11- [17/Jul/2015:16:50:31 -0400] "GET /core HTTP/1.1" 200 3719 0.0034
12- [17/Jul/2015:16:50:31 -0400] "POST /sentiment/bad? HTTP/1.1" 200 - 0.0024
13- [17/Jul/2015:16:50:31 -0400] "GET / HTTP/1.1" 200 4449 0.0086
14- [17/Jul/2015:16:50:31 -0400] "POST /decline_replay HTTP/1.1" 302 - 0.0020
And 30 more exactly at [17/Jul/2015:16:50:31 -0400]
( get /update are longpolling requests so it can take up to 40seconds)
Everythings stops for 12seconds between 10 and 11. And all the requests received during this time seems to be processed simultaneously.
I start the app that way
thin start -p 80
Can it be a thin issue ?
Do i need a custom thin config file ?
Do i need nginx?
Any indication is welcome...
edit :
Errors I find in ObectSpace [SystemStackError, 1][NoMemoryError, 1][IOError, 1]
The kind of behavior smells a lot like request queuing, which means there aren't enough web processes free to handle incoming requests. So the requests sit waiting, and then then the backlog is cleared they suddenly all get processed super fast and all at once.
This guy wrote up a good post on how to use Thin, EventMachine, and Async Sinatra to handle long-polling requests.
I have an annoying problem in Apache Karaf 3.0.2. I want to add a feature repository. It is located in my private Nexus Snapshot Repository and protected by username and password. Everything is correctly stated in my settings.xml
~/.m2/repository/settings.xml:
<servers>
<server>
<id>test</id>
<username>karaf</username>
<password>karafpass</password>
</server>
</servers>
org.ops4j.pax.url.mvn.cfg:
org.ops4j.pax.url.mvn.repositories= \
http://repo1.maven.org/maven2#id=central, \
http://repository.springsource.com/maven/bundles/release#id=spring.ebr.release, \
http://repository.springsource.com/maven/bundles/external#id=spring.ebr.external, \
http://zodiac.springsource.com/maven/bundles/release#id=gemini, \
http://repository.apache.org/content/groups/snapshots-group#id=apache#snapshots#noreleases, \
https://oss.sonatype.org/content/repositories/snapshots#id=sonatype.snapshots.deploy#snapshots#norelease, \
https://oss.sonatype.org/content/repositories/ops4j-snapshots#id=ops4j.sonatype.snapshots.deploy#snapshots#noreleases, \
http://192.168.178.25:8081/nexus/content/repositories/snapshots#id=test#snapshots#noreleases
The command to add the feature repository:
feature:repo-add mvn:com.test/com.test.feature/1.0.0-SNAPSHOT/xml/features
In the first try I get:
Error executing command: Error resolving artifact com.test:com.test.feature:xml:features:1.0.0-SNAPSHOT:
Could not find artifact com.test:com.test.feature:xml:features:1.0.0-SNAPSHOT in apache (http://repository.apache.org/content/groups/snapshots-group/)
Why does it fail? The error statement is right, but why is the artifact not found in my Snapshot-Repository? Here is what I can see in my Nexus request.log:
192.168.178.25 - - [25/Dez/2014:15:44:40 +0100] "GET /nexus/content/repositories/snapshots/com/test/com.test.feature/1.0.0-SNAPSHOT/maven-metadata.xml HTTP/1.1" 401 0
192.168.178.25 - - [25/Dez/2014:15:44:43 +0100] "GET /nexus/content/repositories/snapshots/com/test/com.test.feature/1.0.0-SNAPSHOT/com.test.feature-1.0.0-SNAPSHOT-features.xml HTTP/1.1" 401 0
192.168.178.25 - - [25/Dez/2014:15:44:43 +0100] "GET /nexus/content/repositories/snapshots/com/test/com.test.feature/1.0.0-SNAPSHOT/com.test.feature-1.0.0-SNAPSHOT-features.xml HTTP/1.1" 404 1289
The problem is the 401 in the first line which hinders karaf from getting maven-metadata.xml to form the subsequent commands right. Why? All credentials are part of my M2 settings.xml, just use them!
The next try with the same command succeeds and the request.log says:
192.168.178.25 - - [25/Dez/2014:15:59:02 +0100] "GET /nexus/content/repositories/snapshots/com/test/com.test.feature/1.0.0-SNAPSHOT/maven-metadata.xml HTTP/1.1" 200 1029
192.168.178.25 - - [25/Dez/2014:15:59:02 +0100] "GET /nexus/content/repositories/snapshots/com/test/com.test.feature/1.0.0-SNAPSHOT/maven-metadata.xml.sha1 HTTP/1.1" 200 40
192.168.178.25 - - [25/Dez/2014:15:59:03 +0100] "GET /nexus/content/repositories/snapshots/com/test/com.test.feature/1.0.0-SNAPSHOT/com.test.feature-1.0.0-20141224.100146-1-features.xml HTTP/1.1" 200 1086
192.168.178.25 - - [25/Dez/2014:15:59:04 +0100] "GET /nexus/content/repositories/snapshots/com/test/com.test.feature/1.0.0-SNAPSHOT/com.test.feature-1.0.0-20141224.100146-1-features.xml.sha1 HTTP/1.1" 200 40
Please, I require your help? How can I get rid of that problem?
Is there a way to make and external call once a worflow is completed in Oozie. Reading some documentation it seems like possible. (Oozie can make HTTP callback notifications on action start/end/failure events and workflow end/failure events.)
http://archive.cloudera.com/cdh/3/oozie/WorkflowFunctionalSpec.html
But not sure how to do it and can it be an external api call on callback.
Thanks
Further down the document, section 5
What do you mean by external API call? It can only do an HTTP get - so if you configure a servlet endpoint or something similar to trigger that external API call when the URL is visited, then you should able to do pretty much anything you want.
It is also worth noting however that the docs say:
Oozie will make a best effort to deliver the notifications, in case of failure it will retry the notification a pre-configured number of times at a pre-configured interval before giving up.
There are two properties you can add to your oozie job.properties. The variables $jobId , $status and $nodeName are replaced at runtime.
1. oozie.wf.workflow.notification.url=http://yourserver.com/some/path/$jobId/$status
This will give you a HTTP GET call at http://yourserver.com/some/path/ at job level for each status change for example
myhost.softlayer.com - - [19/Feb/2016 10:35:02] "GET /dbgen/lineitem2/0000050-151213000221460-oozie-oozi-W/RUNNING HTTP/1.1" 200 -
myhost.softlayer.com - - [19/Feb/2016 10:35:35] "GET /dbgen/lineitem2/0000050-151213000221460-oozie-oozi-W/SUCCEEDED HTTP/1.1" 200 -
This one gives more detailed transition between your oozie job steps
2. oozie.wf.action.notification.url=http://yourserver.com/some/path/$jobId/$nodeName/$status
Sample output at server side :
myhost.softlayer.com - - [19/Feb/2016 10:35:02] "GET /dbgen/lineitem2/0000050-151213000221460-oozie-oozi-W/:start:/T:ReadConfig HTTP/1.1" 200 -
myhost.softlayer.com - - [19/Feb/2016 10:35:03] "GET /dbgen/lineitem2/0000050-151213000221460-oozie-oozi-W/ReadConfig/S:RUNNING HTTP/1.1" 200 -
myhost.softlayer.com - - [19/Feb/2016 10:35:09] "GET /dbgen/lineitem2/0000050-151213000221460-oozie-oozi-W/ReadConfig/T:CheckStep HTTP/1.1" 200 -
myhost.softlayer.com - - [19/Feb/2016 10:35:10] "GET /dbgen/lineitem2/0000050-151213000221460-oozie-oozi-W/sqoop-table-import/S:RUNNING HTTP/1.1" 200 -
myhost.softlayer.com - - [19/Feb/2016 10:35:10] "GET /dbgen/lineitem2/0000050-151213000221460-oozie-oozi-W/CheckStep/T:sqoop-table-import HTTP/1.1" 200 -
myhost.softlayer.com - - [19/Feb/2016 10:35:29] "GET /dbgen/lineitem2/0000050-151213000221460-oozie-oozi-W/javaStatsReporting/S:RUNNING HTTP/1.1" 200 -
myhost.softlayer.com - - [19/Feb/2016 10:35:29] "GET /dbgen/lineitem2/0000050-151213000221460-oozie-oozi-W/sqoop-table-import/T:javaStatsReporting HTTP/1.1" 200 -
myhost.softlayer.com - - [19/Feb/2016 10:35:35] "GET /dbgen/lineitem2/0000050-151213000221460-oozie-oozi-W/end/T:null HTTP/1.1" 200 -
myhost.softlayer.com - - [19/Feb/2016 10:35:35] "GET /dbgen/lineitem2/0000050-151213000221460-oozie-oozi-W/javaStatsReporting/T:end HTTP/1.1" 200 -