What would cause SmartCloud to redirect a REST call to a login page? - ibm-sbt

I am not using the SBT, but making direct REST calls with Abdera to the current version of Connections on IBM SmartCloud. REST URL in question: https://apps.na.collabserv.com/search/serviceconfigs
Observations
When testing from my laptop (using Firefox and the REST client add-on,) this works as expected. I get back an ATOM feed.
When testing from a server (on a different network,) using the same method (Firefox + REST client,) I get back HTML that is a log-in page.
In addition, I get this same result when I call the URL from a Java program on the same server.
In all cases, I am using the same credentials with basic authentication.
Update: If I log into SmartCloud first, on a separate tab in Firefox on the server, then call the URL as before, from another tab, it works. I get the ATOM feed as desired. Naturally, this is unsuitable as a solution, but I present it as additional information that could lead to an actual solution.
Update: Further testing shows that the local (laptop) log-in exhibits the same behavior as the server. A form-based log-in is required from the same browser, then subsequent REST calls work.
Update: Here is a relevant simplified code snippet:
private static Abdera ABDERA = new Abdera();
private static AbderaClient ABDERA_CLIENT = new AbderaClient(ABDERA);
...
String host = "https://apps.na.collabserv.com";
ABDERA_CLIENT.addCredentials(host, AuthScope.ANY_REALM, "basic", new UsernamePasswordCredentials("user", "password"));
...
ClientResponse response = ABDERA_CLIENT.get("https://apps.na.collabserv.com/search/serviceconfigs");
Summary
It appears that something about the originating server or the call is causing SmartCloud to respond with a log-in page. Whereas, the same call and credentials from my laptop, work as expected.
Question
Where should I start to trouble-shoot this? How can I build the client credentials to allow programmatic log-in?
Response Headers
If it helps, here are the response headers that I get back in each case.
Unsuccessful
Status Code: 200 OK
Cache-Control: no-cache
Connection: keep-alive
Content-Encoding: gzip
Content-Length: 1850
Content-Type: text/html
Date: Tue, 08 Oct 2013 14:15:03 GMT
Pragma: no-cache
Server: WebSEAL/6.1.1.3 (Build 110428)
Set-Cookie: PD-H-SESSION-ID=4_0_IR4***masked***oRKlJI;secure; Path=/; HttpOnly BIGipServerE3A-WebSEAL-80-fe=2132806922.20480.0000;secure; path=/
Vary: Accept-Encoding
p3p: CP="NON CUR OTPi OUR NOR UNI"
Successful
Status Code: 200 OK
Cache-Control: public, max-age=86400, s-maxage=86400, no-cache=set-cookie, private, must-revalidate
Content-Encoding: gzip
Content-Language: en-US
Content-Length: 1164
Content-Type: application/atom+xml; charset=UTF-8
Date: Mon, 07 Oct 2013 17:21:12 GMT
Expires: Tue, 08 Oct 2013 17:21:12 GMT
Server: WebSphere Application Server/8.0
Vary: Accept-Encoding
p3p: CP="NON CUR OTPi OUR NOR UNI"
x-lconn-auth: true
x-powered-by: Servlet/3.0

#Grant is your login using SAML? I could see this redirect happening. also could be TFIM related... maybe you should grab the auth on a different page, store the cookies, and then try connecting to the endpoint above.

Related

Disable caching in open-uri

I have to, sadly, poll an endpoint and update another system when the data changes. I wrote a loop (with a sleep statement so I don’t DOS the server):
require 'nokogiri'
require 'open-uri'
desired_data = 'foo'
data = nil
url = nil
while data != desired_data do
sleep(2)
url = "https://elections.wi.gov/index.php/elections-voting/statistics"
doc = Nokogiri::HTML.parse(open(url))
puts doc
# do some nokogiri stuff to extract the information I want.
# store information to `data` variable.
end
# if control is here it means the data changed
This works fine except when the server updates, open(url) still returns the old content (even if I restart the script).
It seems like there may be some caching at play. How do I disable it?
Here are the HTTP headers returned:
HTTP/2 200
date: Fri, 02 Oct 2020 14:00:44 GMT
content-type: text/html; charset=UTF-8
set-cookie: __cfduid=dd8fca84d468814dd199dfc08d45c98831601647244; expires=Sun, 01-Nov-20 14:00:44 GMT; path=/; domain=.elections.wi.gov; HttpOnly; SameSite=Lax; Secure
x-powered-by: PHP/7.2.24
cache-control: max-age=3600, public
x-drupal-dynamic-cache: MISS
link: <https://elections.wi.gov/index.php/elections-voting/statistics>; rel="canonical"
x-ua-compatible: IE=edge
content-language: en
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
expires: Sun, 19 Nov 1978 05:00:00 GMT
last-modified: Fri, 02 Oct 2020 12:47:38 GMT
vary: Cookie
x-generator: Drupal 8 (https://www.drupal.org)
x-drupal-cache: HIT
x-speed-cache: HIT
x-speed-cache-key: /index.php/elections-voting/statistics
x-nocache: Cache
x-this-proto: https
x-server-name: elections.wi.gov
access-control-allow-origin: *
x-xss-protection: 1; mode=block
cache-control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
cf-cache-status: DYNAMIC
cf-request-id: 058b368b9f00002ff234177200000001
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
server: cloudflare
cf-ray: 5dbef38c3b6a2ff2-ORD```
If it matters, I’m using Ruby 2.7 on macOS Big Sur.
It might be a problem on the Drupal 8 website itself as it has its own cache manager - and it seems like there's a cache per user somewhere if you have new content using a web browser.
It is easy to see which cache contexts a certain page varies by and which cache tags it is invalidated by: one must only look at the X-Drupal-Cache-Contexts and X-Drupal-Cache-Tags headers!
But those headers are not available in your list. If you're in touch with the website's developers ask them to do the following:
You can debug cacheable responses (responses that implement this interface, which may be cached by Page Cache or Dynamic Page Cache) by setting the http.response.debug_cacheability_headers container parameter to true, in your services.yml. Followed by a container rebuild, which is necessary when changing a container parameter.
That will cause Drupal to send X-Drupal-Cache-Tags, X-Drupal-Cache-Contexts headers.

Disable caching of content in firefox offline mode

I am working on a web application which has user management in place. I find a concerning issue in firefox related to Work Offline. Following are the steps describing the scenario:
User logs in to the application
User performs some action and logs out of the application
If the user now enables Work Offline mode in firefox, he/she can use browser back to access the last page. However, this page is supposed to be secure.
In my opinion this is a data security issue as any other user can apply this technique to fetch valuable information of the last user.
I have used cache control headers to communicate to the browser that HTML content should not be cached. Following are the response headers used:
HTTP/1.1 200 OK
Date: Tue, 05 May 2015 10:39:30 GMT
Server: Apache/2.4.9 (Unix) OpenSSL/0.9.8za
Cache-Control: no-cache, no-store
Expires: Wed, 31 Dec 1969 23:59:59 GMT
Content-Type: text/html;charset=UTF-8
Content-Language: en
Vary: Accept-Encoding
Content-Encoding: gzip
X-Frame-Options: SAMEORIGIN
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
I have used
Cache-Control: no-cache, no-store
Expires: Wed, 31 Dec 1969 23:59:59 GMT
I have noted this vulnerability in applications like Facebook. Is this resolvable? Thank you.

Double Set-Cookie in Magento, leading to a login issue for some users

We have a Magento application which is issuing dual Set-Cookie's . Here are the headers:
HTTP/1.1 200 OK
Date: Wed, 18 Apr 2012 21:04:28 GMT
Server: Apache/2.2.3 (CentOS)
X-Powered-By: PHP/5.2.10
Set-Cookie: frontend=iti6c00cdm6cc79hfl1pl9pq52; expires=Wed, 18-Apr-2012 22:04:28 GMT; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Set-Cookie: frontend=iti6c00cdm6cc79hfl1pl9pq52; expires=Wed, 18-Apr-2012 22:04:28 GMT; path=/; domain=**example.com**
Connection: close
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8
In some circumstances, after logging in the second cookie is set to frontend=deleted . From my reading it appears that two frontend= cookies are not a "problem", this is standard Magento behaviour. From my reading of the spec, the second frontend= cookie will overwrite the first if their scope/spec is the same.
Any ideas where we can start digging in to this problem to see why the second frontend= cookie does not behave like the first?
Magento version is enterprise edition of ver. 1.9.0.0
Related Questions
Why does Magento use 2 cookies per session?
Magento Cookies Changing Prevent Frontend Login
This happens when the Session validation checks fail - the cookie will then be cleared with the "deleted" value and a expiration date in the past:
The following information will be checked by Magento for validating a session:
The client IP address that is connecting to the server
The "Via" HTTP-Header
The "X-Forwarded-For" Header
The "User-Agent" Header
If one (or more) of these informations changes during the requests for the same Session ID, the session will be Discarted, the Cookie will be cleared in the way as described and the Server will send a Redirect header to the Homepage.
You can change which Information to validate in the Magento Admin-Panel by going to System > Configuration > Web. But you should never turn off all checks since this will allow session hijacking.
Do you want to override fronten cookie... if so better try to first destroy the cookie and then reset it by using Magento method
Mage::getModel('core/cookie')->set('frontend', $session->getCustomer()->getId(), 100000*24*3600);

Should I disable WebClient caching?

WebClient.DownloadStringAsync does cache the server response.
After once getting a response from the server I get a response even without internet connection!
Is WebClient caching smart enough to determine from the server response how long to cache?
Or is it buggy and I should disable caching.
Backgound info:
Url: http://www.ecb.europa.eu/stats/eurofxref/eurofxref-daily.xml
Fiddler trace:
GET /stats/eurofxref/eurofxref-daily.xml HTTP/1.1
Accept: /
Referer: file:///Applications/Install/4D0DF1F7-1481-45CA-86BE-C14FF5CCD955/Install/
Accept-Encoding: identity
User-Agent: NativeHost
Host: www.ecb.europa.eu
Connection: Keep-Alive
HTTP/1.1 200 OK
Date: Sun, 25 Mar 2012 08:54:40 GMT
Server: Apache/2.2.3 (Linux/SUSE)
Last-Modified: Fri, 23 Mar 2012 13:31:39 GMT
ETag: "19d4e5-6a9-4bbe90b5904c0"
Accept-Ranges: bytes
Content-Length: 1705
Keep-Alive: timeout=3, max=200
Connection: Keep-Alive
Content-Type: text/xml
Set-Cookie: BIGipServerPOOL.www.ecb.europa.eu_HTTP=2684883628.16415.0000; path=/
...
Disabling caching via Headers does not work:
.Headers("cache-control") = "no-cache"
.Headers("HttpRequestHeader.IfModifiedSince") = DateTime.UtcNow.ToString()
Disabling caching via appending uniqa parameter works:
"http://www.ecb.europa.eu/stats/eurofxref/eurofxref-daily.xml" & "?MakeRequestUnique=" & Environment.TickCount
The integrated cache isn't smart at all. So if you expect different results when querying the page, you have to bypass it. I say 'bypass' because there's no way I know of to disable it with the WebClient (I don't think it's enabled if you directly use the HttpRequest class).
So if you want to use the WebClient, the best way is to append a random parameter to the request.

Ajax callback not called until new tab opens in IE7

We're having some really strange problems with IE7, and only IE7. When trying to replicate the issue it only happens with native IE7, not when running IE7 mode in IE8/9, so please remember that if you try to replicate this issue.
The problem is the following:
We're polling for a response from the server with AJAX. The user posts something that the server may have to work on for a bit, so every 5 seconds or so a request is sent check if the server is done. This works fine in every browser, except native IE7. The problem is that it never stops "loading". When checking the requests with Fiddler2 we see that it does two requests and then nothing more. It stops doing anything when still in PENDING mode. In a sain browser it keeps on polling, and then stops when it gets "CONFIRMED". The really weird thing is that it does its final request and returns as normal, only, and really only, when you open a new tab.
It's not that the page needs focus or anything, clicking around randomly does nothing. I'm asking here because I can't even reproduce the issue using a local instance of the project.
Here are the headers sent back by the server:
These are the headers for the response on the production machine. This was the last response gotten. It loads indefinitely until you open a new tab (just an empty new tab!), then the final request is made and everything works out.
HTTP/1.1 200 OK
Accept-Ranges: bytes
Age: 0
Cache-Control: max-age=0, private, must-revalidate
Content-Type: application/json; charset=utf-8
Date: Tue, 04 Oct 2011 07:37:45 GMT
ETag: "867dafc628c43b6ca8a73d1977669250"
P3P: CP="ALL DSP COR CURa ADMa DEVa OUR IND COM NAV"
Server: nginx/1.0.6
Set-Cookie: _web_session=COOKIE; path=/; expires=Tue, 04-Oct-2011 10:37:45 GMT; HttpOnly
Vary: Accept-Encoding
Via: 1.1 varnish
X-Cache: MISS
X-Runtime: 0.062794
X-UA-Compatible: IE=Edge,chrome=1
X-Varnish: 55900984
Content-Length: 145
Connection: keep-alive
{"direct_publishing_settings_id":9970,"confirmed":"PENDING","errors":{},"username":"************","url":"","blog_id":44606,"platform":"blogg_se"}
These are the headers for the same request on my local server. This does not stall up the requests.
HTTP/1.1 200 OK
X-Ua-Compatible: IE=Edge
Etag: "253c934246a69c9ca821464f80f400b3"
P3p: CP="ALL DSP COR CURa ADMa DEVa OUR IND COM NAV"
Connection: Keep-Alive
Content-Type: application/json; charset=utf-8
Date: Tue, 04 Oct 2011 07:34:22 GMT
Server: WEBrick/1.3.1 (Ruby/1.8.7/2010-01-10)
X-Runtime: 0.459232
Content-Length: 137
Cache-Control: max-age=0, private, must-revalidate
Set-Cookie: _web_session=COOKIE; path=/; expires=Tue, 04-Oct-2011 10:34:22 GMT; HttpOnly
{"direct_publishing_settings_id":10,"confirmed":"PENDING","url":"","blog_id":29,"errors":{},"username":"fsasaffas","platform":"blogg_se"}
If you want to try it you can go to videofy.me, get a new account (it's really easy), when logged in go to videofy.me/blogger/settings/direct_publishing. Choose a blog platform in the first dropdown, press "activate" and write something into the username/password fields that appear, then press the green button and wait forever. After 45 seconds or a random amount of time open a new tab, and see that the request is magically finished.
I'm guessing it has to do something with IE7 cacheing the request, and then something is released if a new tab is opened. But it's just a guess, and googling returns nothing related. I'm posting here because it's so obscure that I hope someone here knows anything about it.

Resources