xcap = very frustrating, put element returns 500 in http 401 in log

[etanrosen]

my friend set up xcap, I checked it out, it seems to be set up properly, "gets" on the whole pres-rules doc, or elements within seem to be working fine, puts on the whole doc seem to be working fine, however, puts on an element within the doc always give me a 401 unauthorized error (in the xcap log). I checked the subscriber table against .xcapclient.ini, they seem to match, The http authentication seems to work, because "get" and "put" on the entire doc are working. any ideas why this aint working? I am plum out of them.

I

[etanrosen]

Here are the methods and results, please bear with me. Number 4 fails.

1.THE GET : (no xpath params)

 https://xcap.sipthor.net/xcap-root@sip2sip.info/pres-rules/users/2233370089/index

RESULT: 200 ok

<?xml version="1.0" encoding="UTF-8"?> <ruleset xmlns="urn:ietf:params:xml:ns:common-policy" xmlns:pr="urn:ietf:params:xml:ns:pres-rules" > <rule id="pres_whitelist"> <conditions> <identity> <one id="sip:myfriend@y.emosip.com" /> </identity> </conditions> <actions> <pr:sub-handling>allow </pr:sub-handling> </actions> <transformations> </transformations> </rule> <rule id="pres_blacklist"> <conditions> <identity> <one id="sip:enemy@y.emosip.com" /> </identity> </conditions> <actions> <pr:sub-handling>block </pr:sub-handling> </actions> <transformations> </transformations> </rule> </ruleset>

2.THE PUT

 https://xcap.sipthor.net/xcap-root@sip2sip.info/pres-rules/users/2233370089/index

(same doc as above)

RESULT= 200 OK

3. THE GET ELEMENT (with xpath param)

 https://xcap.sipthor.net/xcap-root@sip2sip.info/pres-rules/users/2233370089/index/~~/cr:ruleset/cr:rule[@id="pres_whitelist"]/cr:conditions/cr:identity/cr:one[@id="sip:myfriend@y.emosip.com"]?xmlns(cr=urn:ietf:params:xml:ns:common-policy)

RESULT = <one id="sip:myfriend@y.emosip.com" />

... so far, beautiful

4. THE PUT ELEMENT (with xpath param)

 https://xcap.sipthor.net/xcap-root@sip2sip.info/pres-rules/users/2233370089/index/~~/cr:ruleset/cr:rule[@id="pres_whitelist"]/cr:conditions/cr:identity/cr:one[@id="sip:fred@sip2sip.info"]?xmlns(cr=urn:ietf:params:xml:ns:common-policy)

<one id="sip:myfriend@y.emosip.com" />

RESULT: 500 INTERNAL SERVER ERROR

"An error occurred while processing the request. More information is available in the server log."

when I look at this in the log it says 401. Could this mean that openxcap was not authorized to write to mysql ? seems bizzare that this would happen on sip2sip.info

Can someone tell me what is wrong with the last PUT?

[admin]

Your problem is caused by the client. There are others who have experienced the same problem and this is the solution:

http://openxcap.org/ticket/83

[etanrosen]

If you look carefully, there was no programatic solution, only to use a client which had seemed to work around the problem, I am looking for a published answer as to why some puts are allowed and some are not. I actually was one of the respondants on that ticket. Please dont close this ticket, it has not been satisfactorily addressed. Can someone please actually look at my sample gets and puts who is knowledable with the internals of openxcap? I would apprreciate it very much.

[admin]

Do you have the same problem with another XCAP client?

[etanrosen]

I cannot tell with other xcap client and sip2sip.info, the xcap uri is ssl (encrypted) and I cannot tell what they are doing. This method of comparative debugging is not very good anyways, "just look at what the other people are doing", it does not lead to any deeper understanding. Can one of the engineers try my puts and gets and see if he gets the same result? Alternatively, if I can have a non ssl xcap root uri to sip2sip.info, that would help too.

[luci]

First of all, if you get a 500 Internal Server Error, check syslog, you will probably find an error printed there.

As for your example PUT, the URI in the request should be:  https://xcap.sipthor.net/xcap-root@sip2sip.info/pres-rules/users/2233370089/index/~~/cr:ruleset/cr:rule[@id="pres_whitelist"]/cr:conditions/cr:identity/cr:one[@id="sip:myfriend@y.emosip.com"]?xmlns(cr=urn:ietf:params:xml:ns:common-policy)

and the body: <cr:one id="sip:myfriend@y.emosip.com"/>

xcapclient example (which I've tested on sip2sip and works):

# xcapclient --sip-address=user@example.com -ppass --xcap-root=http://xcap.example.com/xcap-root@example.com/ --app pres-rules put '/cr:ruleset/cr:rule[@id="pres_whitelist"]/cr:conditions/cr:identity/cr:one[@id="sip:friend@example.com"]?xmlns(cr=urn:ietf:params:xml:ns:common-policy)'
Reading PUT body from stdin. Type CTRL-D when done
<cr:one id="sip:friend@example.com"/>
201 Created
etag: "xxxx"
content-type: application/xcap-el+xml

[etanrosen]

the "fred@" "myfriend@" difference was a copy and paste error for this ticket, I had been using "myfriend@" both places in the code. The only other difference seems to be that you added a namespace qualifier to the body. I tried this too and it didnt seem to help.

How do I check the syslog on sip2sip.info? can you tell me? it would be most helpful.

Could it be my url escaping, here is the escaped url:

 https://xcap.sipthor.net/xcap-root@sip2sip.info/pres-rules/users/2233370089/index/~~/cr%3aruleset%2fcr%3arule%5b%40id=%22pres%5fwhitelist%22%5d%2fcr%3aconditions%2fcr%3aidentity%2fcr%3aone%5b%40id=%22sip%3amyfriend%40sip2sip.info%22%5d%3fxmlns(cr=urn%3aietf%3aparams%3axml%3ans%3acommon-policy)

unescaped ...

 https://xcap.sipthor.net/xcap-root@sip2sip.info/pres-rules/users/2233370089/index/~~/cr:ruleset/cr:rule[@id="pres_whitelist"]/cr:conditions/cr:identity/cr:one[@id="sip:myfriend@sip2sip.info"]?xmlns(cr=urn:ietf:params:xml:ns:common-policy)

[etanrosen]

two more notes, there is no xml declaration tag in your body. I have tried with and without. Also, xcapclient is a test tool, not going through the http server, can you verify that it works from an external client, i.e., curl?

Thanks ;)

[luci]

Replying to etanrosen:

The only other difference seems to be that you added a namespace qualifier to the body. I tried this too and it didnt seem to help.

You should do that, since the one element is in the common-policy namespace.

How do I check the syslog on sip2sip.info? can you tell me? it would be most helpful.

In the beginning of the ticket you stated "my friend set up xcap"; by that I understood that you have access to an OpenXCAP installation, on which you've had this problem.

Replying to etanrosen:

two more notes, there is no xml declaration tag in your body. I have tried with and without.

The body is not an XML document, but a fragment; the xml declaration processing instruction only makes sense in the beginning of an XML document.

Also, xcapclient is a test tool, not going through the http server, can you verify that it works from an external client, i.e., curl?

I don't really understand this last sentence. What does "xcapclient is a test tool" mean? What HTTP server?

In any case, could you provide me with the *full* command-line invocation of xcapclient that fails? Since it works for me, I need a starting point.

[etanrosen]

ah, good that there's some movement on this issue. I haven't been using xcapclient, I have an application which performs http puts and gets (via curl) to modify the pres-rules doc. I will try again to see if there is anything useful in the openxcap log, so far there hasn't been. failing that, I will try xcapclient again and get back to you. FYI - in the interim, I had set up the ericsson SDS pgm (not to say a bad word) and their xcap host accepts all of my applcation's puts and gets (similar to above), without error.

[etanrosen]

so far no luck, the server log did not tell me anything. now testing with xcapclient and xcap.sipthor.net

1.This fails:

xcapclient --app=pres-rules --account-name=2233370089 -i ./test.txt PUT "/cr:ruleset/cr:rule[@id='pres_whitelist']/cr:conditions/cr:identity/cr:one[@id='sip:myfriend@sip2sip.info']?xmlns(cr=urn:ietf:params:xml:ns:common-policy)" put  https://xcap.sipthor.net/xcap-root@sip2sip.info/pres-rules/users/sip:2233370089@sip2sip.info/index/~~/cr:ruleset/cr:rule[@id='pres_whitelist']/cr:conditions/cr:identity/cr:one[@id='sip:myfriend@sip2sip.info']?xmlns(cr=urn:ietf:params:xml:ns:common-policy) 500 Internal Server Error content-type: text/plain content-length: 96 An error occurred while processing the request. More information is available in the server log. xcap:/var/log/openxcap#

-- CONTD - here is test.txt

<cr:one id="sip:myfriend@sip2sip.info" />

2. This succeeds:

xcapclient --app=pres-rules --account-name=2233370089 GET "/cr:ruleset/cr:rule[@id='pres_whitelist']/cr:conditions/cr:identity/cr:one[@id='sip:myfriend@sip2sip.info']?xmlns(cr=urn:ietf:params:xml:ns:common-policy)"

get  https://xcap.sipthor.net/xcap-root@sip2sip.info/pres-rules/users/sip:2233370089@sip2sip.info/index/~~/cr:ruleset/cr:rule[@id='pres_whitelist']/cr:conditions/cr:identity/cr:one[@id='sip:myfriend@sip2sip.info']?xmlns(cr=urn:ietf:params:xml:ns:common-policy) etag: "2375904a5a8d5775ecc558c4c1f99c33" content-type: application/xcap-el+xml content-length: 38 <one id="sip:myfriend@sip2sip.info" /

3. Can you tell me why?

[etanrosen]

note: the content-type was not set properly on the 'PUT' though it was set properly on the GET. In the help for xcapclient it says that if there is a node selector detected, it will set the content-type to "application/xcap-el+xml. This did not happen, also there does not seem to be a way to manually set the mime type.

[ibc]

Replying to etanrosen:

Could it be my url escaping, here is the escaped url:

 https://xcap.sipthor.net/xcap-root@sip2sip.info/pres-rules/users/2233370089/index/~~/cr%3aruleset%2fcr%3arule%5b%40id=%22pres%5fwhitelist%22%5d%2fcr%3aconditions%2fcr%3aidentity%2fcr%3aone%5b%40id=%22sip%3amyfriend%40sip2sip.info%22%5d%3fxmlns(cr=urn%3aietf%3aparams%3axml%3ans%3acommon-policy)

In the node selector :, @ and / must not be escaped.

unescaped ...

 https://xcap.sipthor.net/xcap-root@sip2sip.info/pres-rules/users/2233370089/index/~~/cr:ruleset/cr:rule[@id="pres_whitelist"]/cr:conditions/cr:identity/cr:one[@id="sip:myfriend@sip2sip.info"]?xmlns(cr=urn:ietf:params:xml:ns:common-policy)

Not valid. [, ] and " must be escaped.

Perhaps the following helps you. It's a method I use in my own XCAP client (Ruby code) to encode the node selector (and it seems to work properly):

def percent_encode(node_selector)
  node_selector.dup.force_encoding('ASCII-8BIT').gsub(/[^a-zA-Z0-9\.,:;\-_?!@$&=+*\/~'()]/){'%%%02x' % $&.ord}
end

[ibc]

I've tested your unescaped URI with my client against an OpenXCAP and don't receive error. This is how the URI should look after percent encoding the node selector:

 https://xcap.sipthor.net/xcap-root@sip2sip.info/pres-rules/users/2233370089/index/~~/cr:ruleset/cr:rule%5b@id=%22pres_whitelist%22%5d/cr:conditions/cr:identity/cr:one%5b@id=%22sip:myfriend@sip2sip.info%22%5d?xmlns(cr=urn:ietf:params:xml:ns:common-policy)

[ibc]

Ops, sure that this bug is the same as I've reported right now: #120

The problem is that the xmlns prefix used in the PUT request with XPATH ("cr" in your case) doesn't match the prefix stored in the server (no one since it's the default namespace).

This is a bug in OpenXCAP, not in the client. And IMHO is critical.

[ibc]

To demostrate this bug, I'm 100% sure (I've already tested it) that the folllowing would work (etanrosen, try it please):

In fact, the following would work:

 https://xcap.sipthor.net/xcap-root@sip2sip.info/pres-rules/users/2233370089/index/~~/cr:ruleset/cr:rule[@id="pres_whitelist"]/cr:conditions/cr:identity/cr:one[@id="sip:myfriend@sip2sip.info"]?xmlns(cr=urn:ietf:params:xml:ns:common-policy)

with body:

<xx:one id="sip:myfriend@y.emosip.com" />

(being xx: the prefix used in the stored document for "urn:ietf:params:xml:ns:common-policy")

Note that this is incorrect since the ns prefix in the query is "cr" different than the prefix used in the body ("xx"). They must match to be a correct request according to spces, but it will worki for OpenXCAP junt in case the stored document uses "xx" prefix. It's obviously a bug in the server.

[ibc]

Ok, good news for OpenXCAP: it's not a bug in the server!

Please read  http://tools.ietf.org/html/rfc4825#section-7.10:

7.10.  Fetch Namespace Bindings

   If a client wishes to insert an element or attribute into a document,
   and that element or attribute is part of a namespace declared
   elsewhere in the document, the client will need to know the namespace
   bindings in order to construct the XML content in the request.  If
   the client has a cached copy of the document, it will know the
   bindings.  However, if it doesn't have the whole document cached, it
   can be useful to fetch just the bindings that are in scope for an
   element, in order to construct a subsequent PUT request.

So the problem here is that the client is performing a PUT operation including a body with ns prefixes not matching the ns used in the stored document. As per the above text this is a bug in the client :)

[ibc]

However, OpenXCAP doesn't implement element namespaces fetching (bug #124) so the client must get all the document, get the namespace prefixes and use them into the body of the PUT when creating/replacing an element.

[ibc]

Let me close this bug as it's a failure in the client as I've explained in my last comments.