Can't Open Binary Stream of file on Sharepoint Documents

I am trying to read stream from a file on Sharepoint Document library. My app has full control on documents. 

OpenBinaryStream throws exception 403. The code works if I use SharePoint Credentials by providing username and password but I want to do this using oauth

I am using following method to access file.

Microsoft.SharePoint.Client.File file = GetTheFileFromTheDocLibrary("expViewSt.xlsx");
                SharePointContextToken contextToken;
                string accessToken;
                Uri sharepointUrl = new Uri(Session["SPHostUrl"].ToString());
                TokenHelper.TrustAllCertificates();
                string contextTokenString = Session["contextToken"].ToString();
                contextToken =
                    TokenHelper.ReadAndValidateContextToken(contextTokenString, Request.Url.Authority);
                accessToken =
                    TokenHelper.GetAccessToken(contextToken, sharepointUrl.Authority).AccessToken;                 
                ClientContext clientContext = TokenHelper.GetClientContextWithAccessToken(sharepointUrl.ToString(), accessToken); 
                FileInformation fInfo = SP.File.OpenBinaryDirect(clientContext, file.ServerRelativeUrl);