Ben Biddington

Whatever it is, it's not about "coding"

Archive for April 2010

Whose leg do I have to hump to get a Facebook developers forum account?

with 4 comments

How does one go about registering for the Facebook developer forum?

Filling out the form

Produces “Could not open socket” error in Chrome and Firefox.

Facebook Connect

On the register screen there is a Facebook Connect option, but this does nothing in Chrome, and does not appear at all in Internet Explorer.

Firefox does produce the window, so I select yes or whatever but then I can still not sign in.

Then I ask for my password to be sent, only to be told there is no account for my address.

Okay, then I think perhaps I don’t need to sign in to access the forum. I sign in to facebook and revisit: still can’t do anything but read posts.

In other news

Interestingly I do have punBB listed on my Application settings screen.


Written by benbiddington

29 April, 2010 at 13:37

Posted in development

Tagged with , , , ,

2010-04-04 — new signature

leave a comment »

My customary signature has officially changed from:

<bb />

to:

<bb>
    <mb />
</bb>

Many thanks and kind regards.

Written by benbiddington

28 April, 2010 at 13:37

Posted in development, diary

Tagged with , , ,

Adobe Content Server — packaging large files is painful

with 5 comments

Ordinarily, working with Adobe Content Server (ACS) is more or less tolerable, but recently we have encountered what may be another indication of the quality of this product.

Packaging

Packaging an ebook amounts to posting a signed request to the server, describing the file you’d like to ingest.

You can use UploadTestJar (which may have its own issues), or (if you’re lucky) you can rewrite some sample codes in your chosen language and use that.

You’d think then, that once you’ve got it working you can summarily be on your way, forget about ACS and finish your application.

And you can.

Until the OutOfMemoryExceptions start

While test-driving our application, we naturally wanted to describe what happens with different sizes of files, so we tried some large ones. These would fail with errors about being out of heap space, errors like:

21-Apr-2010 13:37:26 org.apache.catalina.core.StandardWrapperValve invoke
at com.adobe.adept.servlet.AdeptServlet.doPost(AdeptServlet.java:180)
SEVERE: Servlet.service() for servlet Package threw exception
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Unknown Source)
at java.lang.AbstractStringBuilder.expandCapacity(Unknown Source)
at java.lang.AbstractStringBuilder.append(Unknown Source)
at java.lang.StringBuffer.append(Unknown Source)
at com.adobe.adept.xml.XMLAbstractDigestSink.characters(XMLAbstractDigestSink.java:133)
at com.adobe.adept.xml.XMLSink.characters(XMLSink.java:261)
at com.adobe.adept.xml.XMLFieldReader.characters(XMLFieldReader.java:447)
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.characters(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)
at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown Source)
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Unknown Source)
at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
at com.adobe.adept.servlet.AdeptServlet.doPost(AdeptServlet.java:180)

We tried adjusting the available memory for “sags” (as Dan Rough would put it), and this did work to a certain degree, but is not a satisfactory solution.

To me it looks a bit like an attempt to load entire file into memory at once. Surely this can’t be right, can it?

Examining AdeptServlet

In an effort to understand the nature of the problem before solving it, we decided to have a look at that servlet. We decompiled it and set about finding that class, and it is:

com.adobe.adept.packaging.servlet.Package

In the doPost method, there are these lines:

if (paramParsedRequest.data != null) {
    localObject1 = new PDFPackager(paramParsedRequest.data);
} else {
    localObject1 = new PDFPackager(new File(paramParsedRequest.dataPath));
}

This shows the two methods of loading a PDFPackager.

Examining PDfPackager, we can see the ctor has four overloads including these two:

public PDFPackager(byte[] paramArrayOfByte) throws Exception {
    this(new ByteBufferByteReader(paramArrayOfByte));
}

public PDFPackager(File paramFile) throws Exception {
    this(new FileInputStream(paramFile));
}

So, it appears the problem may result from usage of the first version.

That  paramParsedRequest argument to doPost is of type ParsedRequest, and its data property is a Byte array.

This could be a problem: when submitting a package request with a data node instead of a dataPath node, we’re using the byte array overload.

Where is the error actually coming from?

From the stacktrace it looks as though it is coming from whatever is creating the arguments to supply to Package.doPost.

This is the responsibility of Package‘s supertype: AdeptServlet<RequestParser>. It is this class that is responsible for parsing the http request into one of those ParsedRequest objects, and then supplying that to Package.doPost.

The problems starts here at the top level request handling:

// AdeptServlet<RequestParser>
doPost(HttpServletRequest paramHttpServletRequest, HttpServletResponse paramHttpServletResponse)

This is where the request parsing happens, and then — as the stack trace shows — an error ends up resulting from XMLAbstractDigestSink.characters.

XMLAbstractDigestSink.characters attempts to append data to an internal StringBuffer.

Summary

This mechanism has not been designed in any kind of scalable manner — buffering files in memory is utterly nuts.

Why not just write the posted data to a temp file and use the other PDFPackager ctor?

The solution

Well, one suggestion is to not post the files at all, but make a slightly different packaging request that supplies a path to a file on disk rather than the file itself.

To do so requires — as described in ContentServer_Technical_Reference.pdf — supplying a dataPath node in your request instead of a data node.

The downside for us is that now we need to manage this shared file location — a non-trivial task when working with Windows services.

Another (unlikely) solution

Modify the application, i.e., AdeptServlet<RequestParser> so it first copies the posted file to disk, and then proceeds as though it received a dataPath request.

Pretty hard without the source — it’s probably actually against the law, is it?

References

Written by benbiddington

27 April, 2010 at 14:00

Windows services and net use

leave a comment »

We have some Windows services that need to access network shares, and even though we have net used, those resources are still unavailable. It appears this is because our services are running as LocalSystem.

How to check the connections available to LocalSystem

1. Open command prompt as LocalSystem

Follow these instructions to get a LocalSystem cmd prompt using at.exe.

Note: You can use at.exe only when the Schedule service is running, to find out:

sc query schedule

2. List connections

net use

You will see the set of connections available.

Note this set is different to the list generated by ordinary command prompt (your account).

How to add connection for LocalSystem

Don’t know, that method is not very automatable.

References

Written by benbiddington

27 April, 2010 at 13:37

Posted in development

Tagged with , , , , ,

Async operations and exceptions

leave a comment »

We have had the case where we’re creating a class that allows clients to block while internally it reads an entire stream asynchronously. This class encapsulates the state required to perform such a task.

While attempting to write unit tests for exceptions, we found that an exception thrown during the asynchronous operation would not be thrown to client. Debugging showed that the exception was being thrown, but no notification was being sent to the parent thread.

No such thing as unhandled exceptions on managed threads

[MSDN] [since .NET Framework v2.0] There is no such thing as an unhandled exception on a thread pool [or finalizer] thread. When a task throws an exception that it does not handle, the runtime prints the exception stack trace to the console and then returns the thread to the thread pool.

Errors raised on a child thread are essentially lost when the thread exits. This means there is some work required to propagate these exceptions.

This requires a blocking wait on the part of the client, and a mechanism for storing the exception so the parent thread can read it.

As an example, we have implemented an AsyncStreamReader which contains a blocking ReadAll method. If an asynchronous read fails with an exception, that exception is exposed internally as a field, and the waiting thread is then signalled. Once the waiting thread wakes up it checks the exception field and throws it if required.

We have blocking Read operation that waits for an async read to complete. The notification mechanism is a ManualResetEvent (WaitHandle).

  1. T1: Invoke ReadAll.
  2. T1: Start async operation (spawns T2).
  3. T1: Wait.
    1. T2: Async operation encounters exception.
    2. T2: Store exception in _error field.
    3. T2: Signals T1.
    4. T2: Returns without triggering any subsequent reads.
    5. T2: Thread exits
  4. T1: Parent thread resumes (still inside ReadAll).
  5. T1: Checks _error field. If it is not null, throw it, otherwise return.
  6. T1: Exception is now propagated

References

Written by benbiddington

27 April, 2010 at 13:37

Scala — Futures

leave a comment »

A future is a placeholder for the return value of an asynchronous operation, it’s left to clients to decide when to block and wait for reply value.

It is an alternative to blocking on receive.

For example, the double-bang on Actor causes operation to return a future:

   /**
   * Sends msg to this actor and immediately
   * returns a future representing the reply value.
   */
  def !!(msg: Any): Future[Any] = {
    val ftch = new Channel[Any](Actor.self)
    send(msg, ftch)
    new Future[Any](ftch) {
      def apply() =
        if (isSet) value.get
        else ch.receive {
          case any => value = Some(any); any
        }
      def isSet = value match {
        case None => ch.receiveWithin(0) {
          case TIMEOUT => false
          case any => value = Some(any); true
        }
        case Some(_) => true
      }
    }
  }

Which can then be used to obtain the reply.

The Future class takes an InputChannel as its ctor argument. This channel is monitored to determine the future’s completion status.

In this instance, the future by !! is configured with the reply channel as supplied to send. In short, the actor has sent itself a message and specified that the future’s channel should be notified when complete. The future then just monitors that channel for the reply.

Note: Actor.send invokes the act method using a Reaction, which spawns threads and runs actors.

Only the actor creating an instance of a Channel may receive from it. This means that the future here must be running on the same thread as the actor that created it.

The send call is instructing the reply to be returned to the channel being monitored by the future.

Why does future block until actor returns value?

This is because it blocks on the channel, waiting for reply:

...
def apply() =
    if (isSet) value.get
    else ch.receive {
        case any => value = Some(any); any
    }
...

and Channel.receive is a ultimately a blocking operation, since it invokes receive on the actor it belongs to:

...
def receive[R](f: PartialFunction[Msg, R]): R = {
    val C = this.asInstanceOf[Channel[Any]]
    recv.receive {
        case C ! msg if (f.isDefinedAt(msg.asInstanceOf[Msg])) => f(msg.asInstanceOf[Msg])
    }
}
...

Note that recv here is the Actor supplied in Channel ctor.

Consider this example:

val aFuture = future[String] {
    currentThreadId
};

Internally, a new actor is created and has its double bang invoked:

def future[T](body: => T): Future[T] = {
    case object Eval
    val a = Actor.actor {
        Actor.react {
            case Eval => Actor.reply(body)
        }
    }
    a !! (Eval, { case any => any.asInstanceOf[T] })
}

And by examining the apply method above, we know this blocks until a message is received from channel.

Written by benbiddington

24 April, 2010 at 13:37

Posted in development

Tagged with , , ,

Facebook Graph API — getting access tokens

with 73 comments

As described in the documentation it’s a fairly easy process — and it does not require any signatures.

For example, I have an application with an id 116122545078207, and I am using the URL of this blog (https://benbiddington.wordpress.com) to collect request tokens.

[update, 2010-04-29]

If ever there was a lesson to read specification and documentation carefully, this is it. Thanks to comments from Joshua Inkenbrandt, Alex and Gene Leybzon I realise why my examples don’t work as expected: I have been trying to use a mixture of web server and client_cred authentication flow.

As Gene rightly points out, I should have been using user_agent.

User-Agent Flow (with a web browser)

Following the instructions as specified in section 3.5.1.1., Client Requests Authorization, of the specification, this is a one-step process:

Open this in a browser:

https://graph.facebook.com/oauth/authorize?
    type=user_agent&
    client_id=116122545078207&
    redirect_uri=http%3A%2F%2Fbenbiddington.wordpress.com&
    scope=user_photos,email,user_birthday,user_online_presence

Note: there are several options for scope. These are called extended permissions.

Note: unless you specify offline_access, your tokens will expire as soon as the user signs out of facebook.

Note: client_secret is not supplied:

[3.5.1.  User-Agent Flow] This user-agent flow does not utilize the client secret since the client executables reside on the end user’s computer or device which makes the client secret accessible and exploitable.

You’ll be redirected to:

https://benbiddington.wordpress.com/#access_token=
    116122545078207|
    2.1vGZASUSFMHeMVgQ_9P60Q__.3600.1272535200-500880518|
    QXlU1XfJR1mMagHLPtaMjJzFZp4.

And you have your access token, you can go ahead and use it:

https://graph.facebook.com/me?access_token=
    116122545078207|
    2.1vGZASUSFMHeMVgQ_9P60Q__.3600.1272535200-500880518|
    QXlU1XfJR1mMagHLPtaMjJzFZp4.

According to section 3.5.1. Client Requests Authorization, because we have not supplied the optional secret_type:

secret_type
    OPTIONAL. The access token secret type as described by Section 5.3.
    If omitted, the authorization server will issue a bearer token
    (an access token without a matching secret) as described by Section 5.2.

we have been issued a bearer token.

I think this refers to OAuth 1.0-style authentication using token secret. You’d only need one of those if you were requiring signed requests. This seems to contradict the part above about storing client secret on user agent.

Refreshing tokens

Section 3.5.1 describes that the access token may be delivered with an optional refresh_token fragment. On expiry, this token can be exchanged at the for a new access token. No refresh token is supplied by the Facebook API under User-Agent flow, meaning you’ll have to ask users to sign in again.

Using access tokens

Tokens with no session part

In some cases, like when using 3.7.1.  Client Credentials Flow, you’re issued a token with a missing session part.

Instead of this:

    116122545078207|
    2.1vGZASUSFMHeMVgQ_9P60Q__.3600.1272535200-500880518|
    EyWJJYqrdgQgV1bfueck320z7MM.

you get this:

116122545078207|EyWJJYqrdgQgV1bfueck320z7MM.

These do work in some cases, but are rejected by some resources, for example:

https://graph.facebook.com/me?access_token=116122545078207|EyWJJYqrdgQgV1bfueck320z7MM.

returns  error:

{
   "error": {
      "type": "QueryParseException",
      "message": "An active access token must be used to query information about the current user."
   }
}

this is the same error you get when you request the same resource without supplying a token at all:

http://graph.facebook.com/me

Note that these tokens do work against real resource identifer, i.e., without the me alias. For example, here I can use it against me (benbiddington).

https://graph.facebook.com/benbiddington?access_token=116122545078207|EyWJJYqrdgQgV1bfueck320z7MM.

So the me alias only works in the case where we have a full token — the session part is required.

These tokens also work for accessing your insights (see analytics section):

https://graph.facebook.com/app_id/insights?access_token=116122545078207|EyWJJYqrdgQgV1bfueck320z7MM.

This is described in section 3.7.  Autonomous Client Flows:

Autonomous client flows are used to grant client access to protected resources controlled by the client (i.e. the client is the resource owner). For example, these flows are useful when a service provides both client-specific resources in addition to end user resources.

And more specifially, the Client Credentials Flow is described  in section 3.7.1:

The client credentials flow is used when the client acts autonomously without acting on behalf of a separate resource owner. The client secret is assumed to be high-entropy since it is not designed to be memorize by an end user.

Where a client is:

An HTTP client capable of making authenticated requests for protected resources using the OAuth protocol. [This is third-party application that wants to access a resource owner’s Facebook account.]

And a resource owner:

An entity capable of granting access to a protected resource. [This is the user who owns the Facebook account.]

[TBD: So what?]

Tokens, sessions and that

You can see more information about authentication flow by using a bogus redirect_uri, i.e., one that does not match the Connect URL setting in your application, e.g.:

https://graph.facebook.com/oauth/authorize?
    client_id=116122545078207&
    redirect_uri=http%3A%2F%2Flocalhost&
    scope=user_photos

Executing this gives error:

{
   "error": {
      "type": "OAuthException",
      "message": "Invalid redirect_uri: The Facebook Connect cross-domain
          receiver URL (http://localhost) must have the application's
          Connect URL (https://benbiddington.wordpress.com) as a prefix.
          You can configure the Connect URL in the Application Settings Editor."
   }
}

But there is some information in the query string, that when decoded looks like this:

https://graph.facebook.com/oauth/authorize_success?
    client_id=116122545078207&
    redirect_uri=http://localhost&
    scope=user_photos&
    type=web_server&
    perms=user_photos&
    selected_profiles=500880518&
    session={
        "session_key":"2.vHAZRg0Ac4Dtzm2xiVwXoA__.3600.1272286800-500880518",
        "uid":500880518,
        "expires":1272286800,
        "secret":"vHAZRg0Ac4Dtzm2xiVwXoA__",
        "sig":"7a6fc887240884de883a21e2a2aec3e0"
    }

That session_key:

2.vHAZRg0Ac4Dtzm2xiVwXoA__.3600.1272286800-500880518

looks familiar, it’s the same as the code parameter used in web server authentication flow (Section 3.5.2), and it’s the same pattern as the second segment of an access token.

2.{secret}.3600.{expires_at_seconds_after_epoch}-{user_id}

Where are my extended permissions?

It appears there is some problem with authorizing extended permissions.

For example, when I use the link in step (1) up there, I am prompted with the following screen:

That looks like the set I asked for, and so I select Allow.

But when I then inspect the extended permissions in my Application settings, all I see is this:

For some reason the only extra permission I have is email. And actually, did I even ask for Publish recent activity?

What is wrong here?

Troubleshooting

I can’t see my application in my Application settings screen

Make sure to choose Authorised from the show list on the Application settings screen.

The default view is Recently used which — certainly in my case — does not produce my application.

How do I know what permissions my application has?

Once you find the application in your Application settings list, press Edit settings and then select the Additional Permissions tab on the resultant dialog.

How do I de-authorise an application?

From your Application settings list, press the “x” button the right end of the row next to Application Profile link.

My access tokens only seem valid while a user is signed-in to facebook

You need to request offline_access permissions, e,g,:

https://graph.facebook.com/oauth/authorize?
    type=user_agent&
    client_id=116122545078207&
    redirect_uri=http%3A%2F%2Fbenbiddington.wordpress.com&
    scope=user_photos,email,user_birthday,user_online_presence,offline_access

otherwise your access tokens will expire as soon as the user signs out and you’ll get (at least with client):

{
    "error": {
        "type": "OAuthException",
        "message": "Error processing access token."
    }
}

References

Written by benbiddington

23 April, 2010 at 13:37

Posted in development

Tagged with , , , ,