server response cut off. no idea why
trying to log into a website using Winsock. i submit my data to the login page. it sends a response back with cookies and a redirect to a new url. so i resubmit to the new url, including the cookies i received. but the response i get form that is always strangely cut off (notice the last 3 characters). i have no idea why.
=============== SENT to /?p=login ===============
POST http://www.get5friends.com/?p=login HTTP/1.1
Accept: */*
Referer: http://www.get5friends.com
Accept-Language: en-us
Content-Type: application/x-www-form-urlencoded
Accept-Encoding: gzip , deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; www.ASPSimply.com; Maxthon; .NET CLR 1.1.4322; .NET CLR 1.0.3705)
Host: www.get5friends.com
Content-Length: 58
Connection: close
cache-Control: no-cache
username=gnznroses&password=PASSWORD&remember=on&login=Login
=============== RECEIVED from /?p=login ===============
HTTP/1.1 302
Date: Sun, 12 Feb 2006 06:00:15 GMT
Server: Apache/1.3.33 (Unix) DAV/1.0.3 mod_fastcgi/2.4.2 mod_gzip/1.3.26.1a PHP/4.4.2 mod_ssl/2.8.22 OpenSSL/0.9.7e
Cache-control: private
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
X-Powered-By: PHP/4.4.2
Set-Cookie: PHPSESSID=4b17284460c3d649494327cce4c69079; path=/
Set-Cookie: id_c=3537; expires=Monday, 12-Jun-06 06:00:15 GMT; path=/; domain=www.get5friends.com
Set-Cookie: username_c=gnznroses; expires=Monday, 12-Jun-06 06:00:15 GMT; path=/; domain=www.get5friends.com
Set-Cookie: password_c=44eccf5a12345ba429c3db979fa16; expires=Monday, 12-Jun-06 06:00:15 GMT; path=/; domain=www.get5friends.com
Set-Cookie: email_c=deleted; expires=Saturday, 12-Feb-05 06:00:14 GMT; path=/; domain=www.get5friends.com
Location: http://www.get5friends.com/?p=status
Connection: close
Transfer-Encoding: chunked
Content-Type: text/html
c23
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<title>Get5friends.com - Part of the GetFriends Network</title>
<link href="style.css" rel="stylesheet" type="text/css">
<script language="JavaScript">
function openpopup(popurl){
var winpops=window.open(popurl,"","width=400,height=600,scrollbars")
}
function changeBox(cbox) {
box = eval(cbox);
box.checked = !box.checked;
}
function placeFocus() {
if (document.forms.length > 0) {
var field = document.forms[0];
for (i = 0; i < field.length; i++) {
if ((field.elements[i].type == "text") || (field.elements[i].type == "textarea") || (field.elements[i].type.toString().charAt(0) == "s")) {
document.forms[0].elements[i].focus();
break;
}
}
}
}
</script>
</head>
<body>
<div id="logo"><a href="http://www.get5friends.com"><img src="images/logo.gif" border="0"></a></div>
<div id="cap"></div>
<div id="head">
<div style="float: right; text-align: right; margin-right: 5px;">
Signup » Complete Offer » Refer Friends » Receive Gift</div>
<a href="?p=login&PHPSESSID=4b17284460c3d649494327cce4c69079">» Member Login</a> | <a href="http://www.getfriendsnetwork.com/?page_id=5">Gift Gallery</a> (Now open to Canadians!)</div>
<div id="cap"></div>
<table border="0" width="664" cellpadding="0" cellspacing="0" align="center"><tr><td bgcolor="#86BCAA" valign="top" align="center" style="border: 1px solid #CCCCCC">
<h1>Login</h1></td><tr><td valign="top" class="tdmain"><div align="center">
<form method="post" name="loginForm"><input type="hidden" name="PHPSESSID" value="4b17284460c3d649494327cce4c69079" />
<table width="200" border="0" class="main">
<tr>
<td>Username:</td>
<td><input type="text" name="username" size="20" class="textbox"></td>
</tr>
<tr>
<td>Password:</td>
<td><input type="password" name="password" size="20" class="textbox"></td>
</tr>
<tr>
<td colspan="2" valign="middle"><input type="checkbox" name="remember"><span style="cursor: default" onClick="changeBox('document.loginForm.remember')"> Remember Login Info</span></td>
</tr>
<tr>
<td colspan="2"><input type="submit" value="Login" class="button" name="login"></td>
</tr>
<tr>
<td colspan="2"><a href="http://www.get5friends.com/?p=forgot">Forgot your password?</a></td>
</tr>
</table>
</form>
</div> </td>
</tr>
</table>
<div id="cap"></div>
<div id="head">
<div style="float: right; text-align: right; margin-right: 5px;">
Visit all the sites on our network! <a href="http://getfriendsnetwork.com">GetFriendsNetwork.com</a>
<br><font color="red">NEW! Sign up at our brand-new site: <a href="http://getmanyfriends.com">GetManyFriends.com</a>! </font>
</div>
Copyright © Get5friends.com<br>
<a href="javascript:openpopup('view.php?p=tos');">Terms of Service / FAQ</a> |
<a href="javascript:openpopup('view.php?p=pp');">Privacy Policy</a>
</div>
<div id="cap"></div>
<script language="JavaScript" type="text/javascript" src="wz_tooltip.js"></script>
</body>
</html>
0
=============== SENT to /?p=status ===============
GET http://www.get5friends.com/?p=status HTTP/1.1
Accept: */*
Referer: http://www.get5friends.com
Accept-Language: en-us
Content-Type: application/x-www-form-urlencoded
Accept-Encoding: gzip , deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; www.ASPSimply.com; Maxthon; .NET CLR 1.1.4322; .NET CLR 1.0.3705)
Host: www.get5friends.com
Content-Length: 0
Connection: close
cache-Control: no-cache
Cookie: PHPSESSID=4b17284460c3d649494327cce4c69079; id_c=3537; username_c=gnznroses; password_c=44eccf5a792123453db979fa16; email_c=deleted
=============== RECEIVED from /?p=status (GET) ===============
HTTP/1.1 200 OK
Date: Sun, 12 Feb 2006 06:00:16 GMT
Server: Apache/1.3.33 (Unix) DAV/1.0.3 mod_fastcgi/2.4.2 mod_gzip/1.3.26.1a PHP/4.4.2 mod_ssl/2.8.22 OpenSSL/0.9.7e
Cache-control: private
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
X-Powered-By: PHP/4.4.2
Connection: close
Content-Type: text/html
Content-Encoding: gzip
Content-Length: 1449
what can cause this and how do i solve it?
[7135 byte] By [
gnznroses] at [2007-11-19 18:30:00]
# 1 Re: server response cut off. no idea why
Maybe there is something wrong in your code ;) ? What if you post your code, or at least the part that's responsible for receiving the response?
- petter
# 2 Re: server response cut off. no idea why
the code works on other sites, just not this one:
Private Sub Winsock1_DataArrival(ByVal bytesTotal As Long)
On Error GoTo errexit
Dim strData As String
Dim intBreakPosition As Integer 'blank line start position
Debug.Print "getting data..."
Winsock1.GetData strData
m_strHttpResponse = m_strHttpResponse & strData
If Not m_bHeaderReceived Then
'the HTTP headers is not arrived yet. look for blank line (vbCrlf & vbCrLf)
intBreakPosition = InStr(1, m_strHttpResponse, vbCrLf & vbCrLf)
If intBreakPosition Then
m_bHeaderReceived = True
Headers = Split(Left(m_strHttpResponse, intBreakPosition - 1), vbCrLf)
End If
End If
Exit Sub
errexit:
Debug.Print "error getting data " & Err.Description
Text2.Text = m_strHttpResponse
Text2.Visible = True
End Sub
# 3 Re: server response cut off. no idea why
ok, found the problem i think:
Content-Encoding: gzip
so i found an ocs to unzip the data, but how do i get the data in the first place? winsock just stops receiving it. or, it is receiving it, but the textbox won't display it for some reason (which wouldn't make sense cause you can open a zip or any other file in notepad)
# 4 Re: server response cut off. no idea why
The response is not "cut off". You are receiving it correctly, but you are trying to use it incorrectly. Specifically, you are receiving raw binary data, which might include unprintable characters and (more importantly) might includes valid NULL's, but you are trying to use the data as pure text, which it is not.
Note this in the reply headers:=============== RECEIVED from /?p=status (GET) ===============
HTTP/1.1 200 OK
...
...
Content-Encoding: gzip
Content-Length: 1449
The content-encoding part is telling you that the content has been encoded with gzip compression. A gzip-compressed stream can include valid NULL bytes. NULL bytes will confuse text functions, so that they will believe that they have reached the end of a NULL-terminated charater string.
Don't use text functions when processing the data stream retrieved from an HTTP server. There are many circumstances where the stream can include valid NULL bytes, such as when transmitting a .gif or .jpeg file.
In your particular case, your headers told the server that a gzip-compressed response was acceptable (your header stated "Accept-Encoding: gzip, deflate"). As a fast-fix which might work only for some servers, you can try to force the server to send non-gzip-compressed data by replacing your line that now is "Accept-Encoding: gzip, deflate" with "Accept-Encoding: identity" in your request header, as follows:
=============== SENT to /?p=status ===============
GET http://www.get5friends.com/?p=status HTTP/1.1
Accept: */*
Referer: http://www.get5friends.com
Accept-Language: en-us
Content-Type: application/x-www-form-urlencoded
Accept-Encoding: identity
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; www.ASPSimply.com; Maxthon; .NET CLR 1.1.4322; .NET CLR 1.0.3705)
Host: www.get5friends.com
Content-Length: 0
Connection: close
cache-Control: no-cache
Cookie: PHPSESSID=4b17284460c3d649494327cce4c69079; id_c=3537; username_c=gnznroses; password_c=44eccf5a792123453db979fa16; email_c=deleted
This should force the server to send non-compressed data, although some servers seem to ignore it.
Mike
# 5 Re: server response cut off. no idea why
ok, thanks a lot. changing the accept-encoding to "identity" works.
# 6 Re: server response cut off. no idea why
i'm having a new problem now. it's not related to the previous one, but i don't want to start a new thread. ther's another website i'm logging into, and a lot of times it's returning a status code "408", which is a timeout. however it times out instantly. as soon as i send my request, i get the timeout back in response. i have the code check for the status code and when received try the site again one more time, but i get the same response. sometimes it works fine however. what can cause this?
# 7 Re: server response cut off. no idea why
i'm having a new problem now. it's not related to the previous one, but i don't want to start a new thread.Why not?
To get an answer to your question I think you should contact the webmaster of that site.
- petter
# 8 Re: server response cut off. no idea why
i dind;t want to start a new thread cause i figured people who had come to this one before may come back and see my new problem. and it is kinda related, i mean the response i'm getting is kinda "cut off" just not so literally.
the website i'm logging into is a big site and i know they wouldn't give me time of day with my problem. but considering it always works with a web browser, my guess is something would have to be wrong with the way it's handled in my app. i just don't know what could be wrong to get a timeout request when it's obviously not timing out.
# 9 Re: server response cut off. no idea why
my guess is something would have to be wrong with the way it's handled in my app.Then I suggest you get a good packet sniffer (for HTTP) and then compare the data sent between your app and the server with the data sent between a browser a the server. Then, maybe you'll find something that doesn't match...
- petter
# 10 Re: server response cut off. no idea why
that's the thing tho, i already have. that's actually what i used to get all my header info from. but if it works half the time and not the other half it doesn't really explain it. what confuses me most is that the server gives a timeout error and not some other type of error...
# 11 Re: server response cut off. no idea why
Lots of "big sites" rely on the browser to store cookies and then to return the cookies in subsequent GET requests. Without the cookies, the site will not work.
Are you handling cookies properly, like browsers do? It might explain why the server is refusing to return data to you immediately, without a timeout, since the server can determine immediately the the cookie-less request is defective.
Mike
# 12 Re: server response cut off. no idea why
i do handle cookies, but no cookies are set until after you log in.
here's a log of the request/response:
POST http://pcs.freepay.com/Login.aspx HTTP/1.1
Accept: */*
Referer: http://pcs.freepay.com
Accept-Language: en-us
Content-Type: application/x-www-form-urlencoded
Accept-Encoding: identity
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; www.ASPSimply.com; Maxthon; .NET CLR 1.1.4322; .NET CLR 1.0.3705)
Host: pcs.freepay.com
Content-Length: 93
Connection: close
cache-Control: no-cache
FormAction=Login&Next=&Email=bash@bashsoftware.net&Password=PASS&RememberMe=1&Submit1=Login
=============== RECEIVED from /Login.aspx ===============
HTTP/1.1 408 Request Timeout
Connection: Close
Pragma: no-cache
cache-control: no-cache
Content-Type: text/html; charset=iso-8859-1
# 13 Re: server response cut off. no idea why
i do handle cookies, but no cookies are set until after you log in...
I checked the site, and cookies indeed are set, and they are set before you log in.
I captured the traffic between an ordinary web browser (IE) and the site. Here's what I get in response to the first "GET" request to the site:
GET / HTTP/1.1
Accept: */*
Referer: http://www.dev-archive.com/forum/showthread.php?t=375498
Accept-Language: en-us
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)
Host: pcs.freepay.com
Connection: Keep-Alive
=============================
from pcs.freepay.com
=============================
HTTP/1.1 200 OK
Date: Sat, 18 Feb 2006 08:19:41 GMT
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
P3P: CP="ALL DSP COR CUR CONi OUR SAM STP STA"
X-AspNet-Version: 1.1.4322
Set-Cookie: x=xgh=E&xsp=E; path=/
Set-Cookie: s=sui=EMJ&syu=%7baa%5dN''ddd%26v%5cwxzb_b%26v%5cZ'y%5c_bZ'%60%7b%5cda%7b_xtw%26%5d%7b%5dSaQGKIHML; path=/
Cache-Control: private
Expires: Fri, 17 Feb 2006 15:39:41 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 37439
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "DTD/xhtml1-transitional.dtd">
<html>
... etc ...
So, you can see that there's a significant cookie, at the "Set-Cookie" header.
The cookie is returned to the site when I click on the "Almost done... Continue to Step 2" button. Here's what I get from the capture:
POST /Default.aspx HTTP/1.1
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash, application/vnd.ms-powerpoint, application/vnd.ms-excel, application/msword, */*
Referer: http://pcs.freepay.com/
Accept-Language: en-us
Content-Type: application/x-www-form-urlencoded
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)
Host: pcs.freepay.com
Content-Length: 290
Connection: Keep-Alive
Pragma: no-cache
Cookie: x=xgh=E&xsp=E; s=sui=EMJ&syu=%7baa%5dN''ddd%26v%5cwxzb_b%26v%5cZ'y%5c_bZ'%60%7b%5cda%7b_xtw%26%5d%7b%5dSaQGKIHML; __utma=74704019.318832976.1140250803.1140250803.1140250803.1; __utmb=74704019; __utmc=74704019; __utmz=74704019.1140250803.1.1.utmccn=(referral)|utmcsr=dev-archive.com|utmcct=/forum/showthread.php|utmcmd=referral
HTTPReferer=http%3A%2F%2Fwww.dev-archive.com%2Fforum%2Fshowthread.php%3Ft%3D375498&AffiliateGUID=&ReferralID=0&AffiliateSID=&SignUpCode=&CampaignID=0&CampaignTypeID=0&CreativeID=0&SignUpPathID=196&SID=&Email=bash@bashsoftware.net&Password=PASS&SelectedProductID=392047&Image1.x=281&Image1.y=15
=============================
from pcs.freepay.com
=============================
HTTP/1.1 200 OK
Date: Sat, 18 Feb 2006 08:22:04 GMT
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
P3P: CP="ALL DSP COR CUR CONi OUR SAM STP STA"
X-AspNet-Version: 1.1.4322
Set-Cookie: s=sui=EMJ&syu=%7baa%5dN''ddd%26v%5cwxzb_b%26v%5cZ'y%5c_bZ'%60%7b%5cda%7b_xtw%26%5d%7b%5dSaQGKIHML&ser=2tY%60x&sjr=2tY%60x&sfg=&sdf=D&svb=&sty=&shj=D&sen=D&snm=D&sjk=&szx=ut%60%7bTut%60%7b%60%5cyadt_x%26%5bxa&swe=%3cU%3f%3f&sas=FIEIKJLK&sfv=%40_bx&stg=2tY%60x&sgb=%40_bx&syh=2tY%60x&sqa=%3cU%3f%3f&shn=&saz=D&sws=D&sed=D; path=/
Set-Cookie: User=; path=/
Set-Cookie: x=xgh=E&xsp=E; path=/
Cache-Control: private
Expires: Fri, 17 Feb 2006 15:42:01 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 37645
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "DTD/xhtml1-transitional.dtd">
<html>
<head>
... etc...
So clearly the cookie is being returned to the site.
Again, I think it might be a matter of proper handling of cookies.
Mike
# 14 Re: server response cut off. no idea why
yeah, i took another look last night, and a cookie is indeed set. so my app now handles that. also, i was sending content-type and content-length when using GET which i shouldn't have. i dunno yet if it's fixed the problem tho.
# 15 Re: server response cut off. no idea why
some other problem i have sometimes with that site, is instead of a timeout, it'll keep redirecting me to the same page. like i use a simple GET to get login.aspx and it tells me it's moved to login.aspx...
=============== RECEIVED from /Login.aspx (GetURL) ===============
HTTP/1.1 302 Found
Connection: close
Date: Sun, 19 Feb 2006 00:49:47 GMT
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
P3P: CP="ALL DSP COR CUR CONi OUR SAM STP STA"
X-AspNet-Version: 1.1.4322
Location: http://macminis.freepay.com/Login.aspx
Cache-Control: private
Expires: Sat, 18 Feb 2006 08:09:47 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 155
<html><head><title>Object moved</title></head><body>
<h2>Object moved to <a href='http://macminis.freepay.com/Login.aspx'>here</a>.</h2>
</body></html>
so i check for the Location header, set the url to it (which is the same as it was before in this case) and GET it again, with the same result...
