Deployment in practice

[Octavian Râşniţă]

Hi,

I've recently started to use Mercurial for revision control and the pages
regarding the deployment of a Catalyst app from the Catalyst wiki also help
me very much, but I still don't know what would be the best method for
uploading the files to the server.

How do you do it?
Archive the entire app an dupload to the server then change the permissions
of the files and folders there?
Or upload the modified files one by one?
Or generate a tarball on each revision and unarchive and make, make test it
on the server?
Or something else?

Thank you.

Octavian


_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

[Tobias Kremer-5]

2009/10/9 Octavian Râşniţă <orasnita@...>:
> I've recently started to use Mercurial for revision control and the pages
> regarding the deployment of a Catalyst app from the Catalyst wiki also help
> me very much, but I still don't know what would be the best method for
> uploading the files to the server.
> How do you do it?

We do a checkout of the latest version on our staging server, test
everything and then rsync it to the webservers. Works like a charm.

--Tobias

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

[J. Shirley]

On Fri, Oct 9, 2009 at 7:55 AM, Tobias Kremer <tobias.kremer@...> wrote:

2009/10/9 Octavian Râşniţă <orasnita@...>:

> I've recently started to use Mercurial for revision control and the pages
> regarding the deployment of a Catalyst app from the Catalyst wiki also help
> me very much, but I still don't know what would be the best method for
> uploading the files to the server.
> How do you do it?

We do a checkout of the latest version on our staging server, test
everything and then rsync it to the webservers. Works like a charm.

I have deployment branches, and then push from there.  It's git, but I'd assume you can do the same in hg (though I've never used it)

See: http://blog.woobling.org/2009/05/deployment-branches-with-git.html

Then, you can use this post-hook script on the production machine and simply push to that repo:
http://github.com/perigrin/soho-web/tree/master/hooks/post-update

This works very well, and you can have that be on a deployment server/location and update the working tree of the other servers in the node.

-J

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

[Octavian Râşniţă]

From: "J. Shirley" <jshirley@...>

On Fri, Oct 9, 2009 at 7:55 AM, Tobias Kremer
<tobias.kremer@...>wrote:

> 2009/10/9 Octavian Râşniţă <orasnita@...>:
> > I've recently started to use Mercurial for revision control and the
> > pages
> > regarding the deployment of a Catalyst app from the Catalyst wiki also
> help
> > me very much, but I still don't know what would be the best method for
> > uploading the files to the server.
> > How do you do it?
>
> We do a checkout of the latest version on our staging server, test
> everything and then rsync it to the webservers. Works like a charm.
>
>

I have deployment branches, and then push from there.  It's git, but I'd
assume you can do the same in hg (though I've never used it)

Thank you Tobias and J.

For the moment I think I will use Tobias' way, because I use Mercurial only
for a few days and I don't know if it supports hooks made in bash.

I know that git is made in C and some perl scripts, that it is faster, much
advanced than Mercurial (which is made in Python), but unfortunately it
doesn't work well under Windows, and I do the development under Windows and
the testing and production use under Linux...

Octavian


_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

[John Goulah-3]

2009/10/9 Octavian Râşniţă <orasnita@...>:

> Hi,
>
> I've recently started to use Mercurial for revision control and the pages
> regarding the deployment of a Catalyst app from the Catalyst wiki also help
> me very much, but I still don't know what would be the best method for
> uploading the files to the server.
>
> How do you do it?
> Archive the entire app an dupload to the server then change the permissions
> of the files and folders there?
> Or upload the modified files one by one?
> Or generate a tarball on each revision and unarchive and make, make test it
> on the server?
> Or something else?
>

I like to have a revision per code push and rsync it out.   What I
don't recommend is updating code directly from a repo on the server,
it will get you eventually.

http://blog.johngoulah.com/2009/03/code-deployment-techniques/

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

[Mesdaq, Ali]

Some parts of this message have been removed. Learn more about Nabble's security policy.

I will second this recommendation or something similar to this. It’s a lot easier when you have two branches that serve very specific purposes. One branch is production code and that can be sync’d every few minutes with whatever source control system and with a script similar to the one J Shirley posted a link to the webserver can sync to this code with 100% confidence. The other branch for development is where you check in daily changes. This is the branch you manually have your staging machines sync with. Once it passes the tests the code can be merged into the production branch and will auto deploy. The advantage is you will also have a much easier process in trouble shooting production issues because the changes on the files will be much less so it’s easier to find sources of new issues. Some people like to manually update from production code branches which is the safer way to go but at the very least make sure you have the two separate branches. If your checking things into production to fix typos or similar then you’re not using production branch approach properly.

 

Thanks,

------------------------------------------

Ali Mesdaq (CISSP, GIAC-GREM)

Sr. Security Researcher

Websense Security Labs

http://www.WebsenseSecurityLabs.com

------------------------------------------

 

From: J. Shirley [mailto:jshirley@...]
Sent: Friday, October 09, 2009 9:12 AM
To: The elegant MVC web framework
Subject: Re: [Catalyst] Deployment in practice

 

On Fri, Oct 9, 2009 at 7:55 AM, Tobias Kremer <tobias.kremer@...> wrote:

2009/10/9 Octavian Râşniţă <orasnita@...>:

> I've recently started to use Mercurial for revision control and the pages
> regarding the deployment of a Catalyst app from the Catalyst wiki also help
> me very much, but I still don't know what would be the best method for
> uploading the files to the server.
> How do you do it?

We do a checkout of the latest version on our staging server, test
everything and then rsync it to the webservers. Works like a charm.

I have deployment branches, and then push from there.  It's git, but I'd assume you can do the same in hg (though I've never used it)

See: http://blog.woobling.org/2009/05/deployment-branches-with-git.html

Then, you can use this post-hook script on the production machine and simply push to that repo:
http://github.com/perigrin/soho-web/tree/master/hooks/post-update

This works very well, and you can have that be on a deployment server/location and update the working tree of the other servers in the node.

-J

 

Click here to report this email as spam.

Protected by Websense Hosted Email Security — www.websense.com

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

[kevin montuori]

>>>>> "MA" == Mesdaq, Ali <amesdaq@...> writes:

 MA>  Once it passes the tests the code can be merged into the
 MA>  production branch and will auto deploy.

That's a great solution until someone checks a change into the
production branch accidentally.  As someone already mentioned, it'll
happen eventually.  

I'm a firm believer in not using version control as a deployment tool.
It's more work to create a package, deploy that to test, test, deploy
*the same package* to production, but you at least know that what you
tested is what's in production.  Pulling straight from VC you never
really have that assurance.


k.

--
kevin montuori

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

[J. Shirley]

On Fri, Oct 9, 2009 at 1:18 PM, kevin montuori <montuori@...> wrote:

>>>>> "MA" == Mesdaq, Ali <amesdaq@...> writes:

 MA>  Once it passes the tests the code can be merged into the
 MA>  production branch and will auto deploy.

That's a great solution until someone checks a change into the
production branch accidentally.  As someone already mentioned, it'll
happen eventually.

I'm a firm believer in not using version control as a deployment tool.
It's more work to create a package, deploy that to test, test, deploy
*the same package* to production, but you at least know that what you
tested is what's in production.  Pulling straight from VC you never
really have that assurance.

A lot of these assertions are pretty much FUD.

Even exporting a tarball is sourcing from version control.  You're just bypassing a lot of the built-in aspects of source control and applying manual steps that can be messed up in a variety of ways.

Unless it is sourced from source control in a way you can easily and programmatically detect, you really don't know what is in production  outside of "a tarball".    You can keep an archive of tarballs, but you lose context and history... so what's the win?

If something happens and you need to find out the history, what do you do?  You ask version control!

There are plenty of ways to manage how things get into a production branch.  Then, you simply add in another layer using tags.  After all, that's pretty much why tags exist, and how they're used, in every sufficiently modern system.

You deploy a tag, from a branch, and everybody wins.

Keep in mind that I'm not advocating (nor are the posts I linked to) running from a -managed- tree.  The post-update hook I linked to keeps its own git repository to itself, it isn't share.  You have to explicitly push to it to update.

These things are very hard to get wrong, and they work very well.  You just have to learn the tools sufficiently.

-J

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

[Devin Austin]

On Fri, Oct 9, 2009 at 3:08 PM, J. Shirley <jshirley@...> wrote:

On Fri, Oct 9, 2009 at 1:18 PM, kevin montuori <montuori@...> wrote:

>>>>> "MA" == Mesdaq, Ali <amesdaq@...> writes:

 MA>  Once it passes the tests the code can be merged into the
 MA>  production branch and will auto deploy.

That's a great solution until someone checks a change into the
production branch accidentally.  As someone already mentioned, it'll
happen eventually.

I'm a firm believer in not using version control as a deployment tool.
It's more work to create a package, deploy that to test, test, deploy
*the same package* to production, but you at least know that what you
tested is what's in production.  Pulling straight from VC you never
really have that assurance.

A lot of these assertions are pretty much FUD.

Even exporting a tarball is sourcing from version control.  You're just bypassing a lot of the built-in aspects of source control and applying manual steps that can be messed up in a variety of ways.

Unless it is sourced from source control in a way you can easily and programmatically detect, you really don't know what is in production  outside of "a tarball".    You can keep an archive of tarballs, but you lose context and history... so what's the win?

If something happens and you need to find out the history, what do you do?  You ask version control!

There are plenty of ways to manage how things get into a production branch.  Then, you simply add in another layer using tags.  After all, that's pretty much why tags exist, and how they're used, in every sufficiently modern system.

You deploy a tag, from a branch, and everybody wins.

Keep in mind that I'm not advocating (nor are the posts I linked to) running from a -managed- tree.  The post-update hook I linked to keeps its own git repository to itself, it isn't share.  You have to explicitly push to it to update.

These things are very hard to get wrong, and they work very well.  You just have to learn the tools sufficiently.

-J

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

I'm coming in late but deploying a tag from trunk/etc and "switching" the production directory that nginx/lighttpd/apache is looking at is probably the coolest thing ever.  You have a great deal of control when needing to roll back, and a record of all previous tags/versions.

Anyway, my $0.02.

-Devin


--
Devin Austin
http://www.codedright.net
9702906669 - Cell

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

[Mesdaq, Ali]

Solving the issue of someone accidentally checking something into production branch is a lot easier to solve than all the problems associated with not having version control. To compare which do you think will be an easier issue to solve: Someone accidentally checking something into production branch that breaks stuff or someone making a change to production server with no version control that breaks stuff? At least with the first issue you can roll back the change and get back to stable instantly. With the other issue you could be troubleshooting for a long time.

Thanks,
------------------------------------------
Ali Mesdaq (CISSP, GIAC-GREM)
Sr. Security Researcher
Websense Security Labs
http://www.WebsenseSecurityLabs.com
------------------------------------------


-----Original Message-----
From: kevin montuori [mailto:montuori@...]
Sent: Friday, October 09, 2009 1:19 PM
To: The elegant MVC web framework
Subject: [Catalyst] Re: Deployment in practice

>>>>> "MA" == Mesdaq, Ali <amesdaq@...> writes:

 MA>  Once it passes the tests the code can be merged into the
 MA>  production branch and will auto deploy.

That's a great solution until someone checks a change into the
production branch accidentally.  As someone already mentioned, it'll
happen eventually.  

I'm a firm believer in not using version control as a deployment tool.
It's more work to create a package, deploy that to test, test, deploy
*the same package* to production, but you at least know that what you
tested is what's in production.  Pulling straight from VC you never
really have that assurance.


k.

--
kevin montuori

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/


 Protected by Websense Hosted Email Security -- www.websense.com

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

[Dave Rolsky]

On Fri, 9 Oct 2009, Mesdaq, Ali wrote:

> At least with the first issue you can roll back the change and get back
> to stable instantly. With the other issue you could be troubleshooting
> for a long time.

But I think the OP (other poster) mentioned packages. Presumably you can
roll back to an earlier package as easily as rolling back to an earlier
state of the tree.


-dave

/*============================================================
http://VegGuide.org               http://blog.urth.org
Your guide to all that's veg      House Absolute(ly Pointless)
============================================================*/

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

[Bill Moseley]

2009/10/9 Octavian Râşniţă <orasnita@...>

Hi,

I've recently started to use Mercurial for revision control and the pages regarding the deployment of a Catalyst app from the Catalyst wiki also help me very much, but I still don't know what would be the best method for uploading the files to the server.

How do you do it?
Archive the entire app an dupload to the server then change the permissions of the files and folders there?
Or upload the modified files one by one?
Or generate a tarball on each revision and unarchive and make, make test it on the server?

I have a script on a build server that does an svn export of a specific branch and version  (or latest trunk if no version specified).  The script then runs the test suite, runs the "build" scripts that generate "minified" css, javascript, and anything else that prepares the package such as create a meta file with build info, and then builds a tarball named after the reversion.  Also, a symlink is created as "latest" on the build server.

Then on target machines (testing, staging, and production) I have a "push" script that fetches the tarball (based on a specific version or "latest"), untars it on the machine.  It's untarred into a directory named after the revision it came from.  Tests are run that verify that it can talk to whatever it needs (database, services).  Then do a graceful stop of the existing server, move symbolic links, start the server back up and then use wget to validate a number of test URLs.  Depending on the update some servers may get pulled out of the balancer before upgrading to do it in stages.

Pushing a previous version will just update the symlinks if they revesion exists.

Not perfect but seems to work ok for now.

--
Bill Moseley
moseley@...

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

[Octavian Râşniţă]

From: "Bill Moseley" <moseley@...>
2009/10/9 Octavian Râşniţă <orasnita@...>

> Hi,
>
> I've recently started to use Mercurial for revision control and the pages
> regarding the deployment of a Catalyst app from the Catalyst wiki also
> help
> me very much, but I still don't know what would be the best method for
> uploading the files to the server.
>
> How do you do it?
> Archive the entire app an dupload to the server then change the
> permissions
> of the files and folders there?
> Or upload the modified files one by one?
> Or generate a tarball on each revision and unarchive and make, make test
> it
> on the server?
>

I have a script on a build server that does an svn export of a specific
branch and version  (or latest trunk if no version specified).  The script
then runs the test suite, runs the "build" scripts that generate "minified"
css, javascript, and anything else that prepares the package such as create
a meta file with build info, and then builds a tarball named after the
reversion.  Also, a symlink is created as "latest" on the build server.

Then on target machines (testing, staging, and production) I have a "push"
script that fetches the tarball (based on a specific version or "latest"),
untars it on the machine.  It's untarred into a directory named after the
revision it came from.  Tests are run that verify that it can talk to
whatever it needs (database, services).  Then do a graceful stop of the
existing server, move symbolic links, start the server back up and then use
wget to validate a number of test URLs.  Depending on the update some
servers may get pulled out of the balancer before upgrading to do it in
stages.

Pushing a previous version will just update the symlinks if they revesion
exists.

Not perfect but seems to work ok for now.


Thank you all for your answers. It was very helpful.
What about the database? If starting from a certain revision you need to
make some database changes (adding tables, adding/deleting some table
fields) how can we automate this?

I mean, I think that we could run a script that add some tables and remove
the unneeded fields from other tables, but if something goes wrong and we
need to go back to the previous version, we would need the records from
those fields.

What can we do in these situations? Do the database update and the tests
manually? Or back-up the entire database and then test the latest version of
the app?

Thanks.

Octavian


_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

[dab-4]

> Thank you all for your answers. It was very helpful.
> What about the database? If starting from a certain revision you need
> to make some database changes (adding tables, adding/deleting some
> table fields) how can we automate this?
>
> I mean, I think that we could run a script that add some tables and
> remove the unneeded fields from other tables, but if something goes
> wrong and we need to go back to the previous version, we would need
> the records from those fields.
>
> What can we do in these situations? Do the database update and the
> tests manually? Or back-up the entire database and then test the
> latest version of the app?
>
> Thanks.
>
> Octavian

DBIx::Class::Schema::Versioned perhaps ?

Dab

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

[Octavian Râşniţă]

From: "dab" <dab@...>

>> Thank you all for your answers. It was very helpful.
>> What about the database? If starting from a certain revision you need
>> to make some database changes (adding tables, adding/deleting some
>> table fields) how can we automate this?
>>
>> I mean, I think that we could run a script that add some tables and
>> remove the unneeded fields from other tables, but if something goes
>> wrong and we need to go back to the previous version, we would need
>> the records from those fields.
>>
>> What can we do in these situations? Do the database update and the
>> tests manually? Or back-up the entire database and then test the
>> latest version of the app?
>>
>> Thanks.
>>
>> Octavian
>
> DBIx::Class::Schema::Versioned perhaps ?
>
> Dab

I don't know... Does anyone use it?

I read in its POD docs that "At the moment, only SQLite and MySQL are
supported.", however in the past I've seen that when creating MySQL queries
from a DBIC schema, some fields were wrongly created, including the size
when not needed, like
date date(10),
so they should be corrected manually. Because of this, I don't know if this
tool can be used for do the change automaticly...
(Or was this issue solved since then?)

Octavian





_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

[Ash Berlin-2]

On 11 Oct 2009, at 11:59, Octavian Râsnita wrote:

> From: "dab" <dab@...>
>>> Thank you all for your answers. It was very helpful.
>>> What about the database? If starting from a certain revision you  
>>> need
>>> to make some database changes (adding tables, adding/deleting some
>>> table fields) how can we automate this?
>>>
>>> I mean, I think that we could run a script that add some tables and
>>> remove the unneeded fields from other tables, but if something goes
>>> wrong and we need to go back to the previous version, we would need
>>> the records from those fields.
>>>
>>> What can we do in these situations? Do the database update and the
>>> tests manually? Or back-up the entire database and then test the
>>> latest version of the app?
>>>
>>> Thanks.
>>>
>>> Octavian
>>
>> DBIx::Class::Schema::Versioned perhaps ?
>>
>> Dab
>
> I don't know... Does anyone use it?
>
> I read in its POD docs that "At the moment, only SQLite and MySQL  
> are supported.", however in the past I've seen that when creating  
> MySQL queries from a DBIC schema, some fields were wrongly created,  
> including the size when not needed, like
> date date(10),
> so they should be corrected manually. Because of this, I don't know  
> if this tool can be used for do the change automaticly...
> (Or was this issue solved since then?)
>
> Octavian
>

refactoring

Schema::Versioned uses SQL::Translator under the hood to do its job.  
Work has been done to make SQL::T v1 better, and as part of this years  
Google Summer of Code, Justin Hunter was the internals to make it  
easier to fix problems. So the problems should either be fixed  
already, or be fixed soon.

-ash
_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/

[Bill Moseley]

2009/10/11 Octavian Râşniţă <orasnita@...>

Thank you all for your answers. It was very helpful.
What about the database? If starting from a certain revision you need to make some database changes (adding tables, adding/deleting some table fields) how can we automate this?

Every database update is slightly different, so automating hasn't seemed possible.

The applications includes a script to build a test database.  The test database is saved in svn with each application.   The test files can load this database to test against when running make test.  That stays in sync with development.

I also have a cron job running on a test server that does an svn checkout when the revision number changes (for branches or trunk) and runs tests against this test database.

 

I mean, I think that we could run a script that add some tables and remove the unneeded fields from other tables, but if something goes wrong and we need to go back to the previous version, we would need the records from those fields.

The database is more complex.  We create a set of change files that the DBA applies.  I like hand creating these and filling them with comments on why a column is added or dropped, and the tickets they apply to.  They are in svn (for a history) but also in a web app the DBA uses to track what's been applied and what is scheduled to apply.

It's very rare that a release includes database changes that have to be done at the moment of pushing the new version.  Schema changes happen first.  That is, the database is ready for the new features before the application is pushed.  Sometimes constraints have to be added in after pushing the release.  Bigger changes require a service window, of course.  In those cases I have created a script that makes the database changes, populates tables and runs tests.

Once changes are in the database tests can be run against a snapshot of the production db instead of the test database.

I'd like to have it more automated, but it seems like every database change requires something special.
 

 

--
Bill Moseley
moseley@...

_______________________________________________
List: Catalyst@...
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@.../
Dev site: http://dev.catalyst.perl.org/