AMF-PHP to scrape state "Sunshine Portal"?

[David Collins-11]

Hello,

I'm a journalist and somewhat of a PHP developer. I've been trying to get complete data from the State of New Mexico's official transparency portal for a couple of years. We've negotiated with mixed results for full csv downloads. I've used Flashbug and Charles to dig into the code. I've been provided a proof-of-concept MXML script by a former in-house Adobe developer that demonstrated the ability to retrieve data, but didn't have resources to pay for ongoing development.

What I need is to use a PHP server to retrieve and translate AMF files from that remote server. I have full control of the PHP server, and no control over the server on the other end.

Question: is AMFPHP a viable tool for this requirement?

Question 2: Does anyone here want to correspond with me further about this challenging scrape project? I can explain the journalistic reasons we need to retrieve the raw data. Short answer is, it's public data, and we need to analyze it outside the Flash interface.

David Collins

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
amfphp-general mailing list
amfphp-general@...
https://lists.sourceforge.net/lists/listinfo/amfphp-general

[arielsom]

Hi,
1:
This is completely out of the scope of  the usual use cases for amfphp. But it's doable.
you can use the serializer and deserializer from amfphp to generate AMF requests, send them to their server and parse their answers.
The best tool for that would be an AIR app in Flex, but obviously if you don't know Flex but know PHP it could work for you.
It's a long and difficult job though, I'm warning you.
2:
I can provide some support through the forums.
Ariel Sommeria
(Team lead for amfPHP)

On Wed, May 23, 2012 at 8:45 PM, David Collins <collidavid@...> wrote:

Hello,

I'm a journalist and somewhat of a PHP developer. I've been trying to get complete data from the State of New Mexico's official transparency portal for a couple of years. We've negotiated with mixed results for full csv downloads. I've used Flashbug and Charles to dig into the code. I've been provided a proof-of-concept MXML script by a former in-house Adobe developer that demonstrated the ability to retrieve data, but didn't have resources to pay for ongoing development.

What I need is to use a PHP server to retrieve and translate AMF files from that remote server. I have full control of the PHP server, and no control over the server on the other end.

Question: is AMFPHP a viable tool for this requirement?

Question 2: Does anyone here want to correspond with me further about this challenging scrape project? I can explain the journalistic reasons we need to retrieve the raw data. Short answer is, it's public data, and we need to analyze it outside the Flash interface.

David Collins

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
amfphp-general mailing list
amfphp-general@...
https://lists.sourceforge.net/lists/listinfo/amfphp-general

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
amfphp-general mailing list
amfphp-general@...
https://lists.sourceforge.net/lists/listinfo/amfphp-general

[David Collins-11]

"Doable" sounds good. Long-and-difficult? I'm forewarned. I'll work my way into this and see what I learn before I conclude it's too much effort.

I have Flashbug and SoapUI for testing and analyzing AMF calls. I have a proof-of-concept .mxml written by a former in-house Adobe developer that's demonstrated the feasibility of retrieving data from this site without using the site interface.

Thanks for your reply.

David Collins

On Wed, May 23, 2012 at 2:22 PM, Ariel Sommeria <ariel.publique@...> wrote:

Hi,
1:
This is completely out of the scope of  the usual use cases for amfphp. But it's doable.
you can use the serializer and deserializer from amfphp to generate AMF requests, send them to their server and parse their answers.
The best tool for that would be an AIR app in Flex, but obviously if you don't know Flex but know PHP it could work for you.
It's a long and difficult job though, I'm warning you.
2:
I can provide some support through the forums.
Ariel Sommeria
(Team lead for amfPHP)

On Wed, May 23, 2012 at 8:45 PM, David Collins <collidavid@...> wrote:

Hello,

I'm a journalist and somewhat of a PHP developer. I've been trying to get complete data from the State of New Mexico's official transparency portal for a couple of years. We've negotiated with mixed results for full csv downloads. I've used Flashbug and Charles to dig into the code. I've been provided a proof-of-concept MXML script by a former in-house Adobe developer that demonstrated the ability to retrieve data, but didn't have resources to pay for ongoing development.

What I need is to use a PHP server to retrieve and translate AMF files from that remote server. I have full control of the PHP server, and no control over the server on the other end.

Question: is AMFPHP a viable tool for this requirement?

Question 2: Does anyone here want to correspond with me further about this challenging scrape project? I can explain the journalistic reasons we need to retrieve the raw data. Short answer is, it's public data, and we need to analyze it outside the Flash interface.

David Collins

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
amfphp-general mailing list
amfphp-general@...
https://lists.sourceforge.net/lists/listinfo/amfphp-general

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
amfphp-general mailing list
amfphp-general@...
https://lists.sourceforge.net/lists/listinfo/amfphp-general

--
phone [phone]

Facebook: http://tinyurl.com/dacfbwall

davidacollins.com

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
amfphp-general mailing list
amfphp-general@...
https://lists.sourceforge.net/lists/listinfo/amfphp-general