Userscripts.org in 2008

Posted by Jesse Andrews on Jan 5, 2009

A 2008 summary of site statistics, review of site improvements, and a look at groups and Greasefire. Want a t-shirts?

52,956,864 Pageviews in 2008

almost 53 million page views in 2008

These page views are by real visitors (running JavaScript) and do not include downloads (scripts, images, or other static content), userscript updaters, ajax requests, spiders, and other bot traffic.

Over 1,400,000 dynamic requests per day

average 900+ requests per minute

An average of 900 requests per minute (cpm is calls per minute) hit the rails code running the site (that is over half a billion dynamic requests per year). Pingdom lets me know when the server explodes (which happens at the worst possible times). The site has had less than 3 hours downtime over the last 3 months.

3.5TB downloaded, 0.5TB uploaded

4 TeraBytes transferred

The downstream bandwidth has averaged 971Kb per second for the year (1.4Mb per second since September!). If the site was hosted on Amazon's EC2, bandwidth would cost $445 for the year. I expect the bandwidth to double in 2009.

Constant battles with CPU

the red is bad!

Red is BAD! To combat constant growth, there have been many rewrites and infratstucture improvements. Systems changes include sphinx for search, Phusion Passenger for app servers, Quantcast for public stats, and NewRelic for performance monitoring.

Site Improvements

I cannot stress enough that the help given by community members in the site forum has been invaluable. After a long day working on my startup, their contributions give me the energy to spend more time in front of a computer trying to keep up with them. I'm grateful for all levels of help, from reporting a simple bug, helping newbies in forums, to writing html, css, and userscripts to improve the site.

The forum system was rewritten several times: moving script comments to forums, dealing with performance issues, read/unread status rewrite, stylistic improvements, and more. There were 44,673 posts in 2008 (and that doesn't include all the deleted spam).

The site had several major stylistic changes. The major being a new site header and script page. The year also brought many stylistic improvements from members (suggested in the forum).

Reviews were added in late October as a way to help users find the best scripts. So far 702 have been written (with 331 written in December).

Instead of overwriting the data with each upload, storing older versions of scripts was implemented. This helps answer the question "what changed?" and allows the community to watch for funny business by malicious script authors.

There are 25,386 scripts on the site, of which 15,286 were uploaded/updated in 2008.

The Future (where is my jet pack?)

In 2009 I'm concentrating on features that help users sort through all those scripts.

Steve Krulewitz wrote an open source extension called Greasefire help users know when there are scripts on userscripts.org written for the page they are viewing. Join him in the greasefire forum to discuss this new feature.

I've been working on groups with the goal of making it easier to find, share and collaborate on scripts (the first version released yesterday). A group can be centered around a site (NeoPets, Travian, Orkut, ...), concepts (using xpath), or a place for people to collaborate. The initial version of groups include members, forums and script collections. As the feature matures I hope to feature groups on the site homepage.

I plan on moving the entire site to Amazon's scalable infrastucture as I'm hitting the edge of what a single box can handle.

Last, I'm debating having a "userscripts t-shirts" made. Would you be interested in buying one? (probably $20 plus shipping for a high quality shirts) Let me know if you have opinions on the design or would buy one.

I hope that 2009 is another great year!

You could comment on this post if you were logged in.

3 points
login to vote
Posted Jan 5, 2009

I like the groups idea. T-shirts would be neat too. :)

2 points
login to vote
Posted Jan 5, 2009

Great post, very interesting to read all that information.
You should make a time-line with all the great implementations you've made.

2 points
login to vote
Posted Jan 5, 2009

You did great job on USo last year. You have my thank.

2 points
login to vote
Posted Jan 5, 2009

Thank you for your dedication to the site

2 points
login to vote
Posted Jan 6, 2009

Thanks for everything you've done and plan to do!

1 point
login to vote
Posted Jan 7, 2009

I say the server rolls over and dies by March lol.

2 points
login to vote
Jesse Andrews article's author
Posted Jan 7, 2009

I just got off the phone with softlayer - I just upgraded to a $530/mo server plan (was paying about half that at serverbeach, whose support isn't as good as softlayer) - I hope to move tomorrow.

Having one really good server seems better than a bunch of smaller servers for now, so maybe Amazon's EC2 next year...

0 points
login to vote
Posted Jan 7, 2009

Provided you don't change the site structure dramatically, I bet the pageview count will be at 400,000 by the end of next year.

1 point
login to vote
Posted Jan 7, 2009

Jesse, maybe you should put a paypal "Donate" button somewhere on the site so people can chip in and help you pay for the hosting costs.

-22 points
login to vote
Posted Jan 14, 2009
2 points
login to vote
Posted Jan 15, 2009

Congrats on your success through 2008 and my best hopes to you in the new year!

3 points
login to vote
Posted Jan 20, 2009

Wow, you've been busy! I can't believe how much this place has grown since it first started *nostalgia*. Oh, and I second Yansky's idea for a Donate button. Maybe in the form of one of those "Buy Me a Cup of Coffee" buttons you see all the kids with blogs using these days.

2 points
login to vote
Posted Feb 3, 2009

What about issues currently pending... (around 400 pages... with almost 20 script reports on each!!!) ???

1 point
login to vote
Posted Mar 16, 2009

Yes, reporting issues seems pretty useless if you're not gonna do anything with the report

2 points
login to vote
Posted Mar 26, 2009

Whatever happened to having scripts that contained "cookie" in them automatically unlisted. They would still be added to the "need to be reviewed" box but then it would help protect the unsuspecting user from installing all of those cookie grabbers that have popped up.

1 point
login to vote
Posted Mar 30, 2009

@nungryscpro
there are plenty legit scripts that contain "cookie" and there are many ways to obfuscate calls to document.cookie. so if such a system as you described were put in place only the non-malicious scripts that have no reason to hide their intentions would be unlisted, and malicious scripts would still be listed, creating a false sense of security for users. I mean, how the hell do you suggest we prevent something similar to:

var first = "c";
// Hundreds of lines of code
var steal = document[first + "ookie"];

1 point
login to vote
menexis User
Posted Mar 31, 2009

I though the point of the reports were to use them to solve issues

-11 points
login to vote
relish User
Posted Mar 31, 2009
-11 points
login to vote
relish User
Posted Mar 31, 2009
1 point
login to vote
Posted Apr 1, 2009

@relish: Are you asking us to tell you how to create a cookie-stealing script?

1 point
login to vote
Posted Apr 6, 2009

I understand there are legit scripts that contain "cookie" but that's how the sytem used to be set up. I had one of my scripts initially unlisted and had to get it listed by the staff all because I had to reference "cookie.gif"

I guess since they no longer wish to be bothered by reports, they've allowed cookie stealing script to get listed automatically. Under the old system there were 0 cookie grabbing scripts for keyword "neopets". Now there are 14 that have been uploaded within the last 6 weeks. (Half don't actually work because they were written badly.)

When a script is listed or updated, it should be easy enough to parse the code, not just reading it, but basically running it and then searching the resulting page for "document.cookie". I know there are also scripts that aren't malicious that access "document.cookie" but are there going to be that many that the staff will have to list manually? They already have it set up so that when you visit the source code, the script is parsed for syntax highlighting.

In the end, those 14 or so scripts have all been reported but all of them are still there. Even with a couple warnings in the comments the install counts rise. Someone needs to do something about the reports or set up a system that blocks them in the first place!

1 point
login to vote
Posted Apr 13, 2009

@nungryscpro

I understand there are legit scripts that contain "cookie" but that's how the sytem used to be set up. I had one of my scripts initially unlisted and had to get it listed by the staff all because I had to reference "cookie.gif"

First of all their is no staff. Jesse is the only active admin. This is a community driven site. It is up to those of us who understand javascript to be on the lookout and warn people in comments if a script is malicious and it is up to users to take the time to look at the comments to see if a script has been called malicious. Also the install count isn't entirely truthful. You need to install a script to raise its install count. Hell 4.66% of traffic a month is by those who have IE6.

Under the old system there were 0 cookie grabbing scripts for keyword "neopets".

How many scripts were there for neopets?

I know there are also scripts that aren't malicious that access "document.cookie" but are there going to be that many that the staff will have to list manually?

Yes there would. Jesse is just one man. He would have to sit in front of a computer all day listing scripts. There are literally thousands of legit scripts that use "document.cookie". Hell 3 out of my 16 scripts use it.

They already have it set up so that when you visit the source code, the script is parsed for syntax highlighting.

Highlighting is done on the client-side. Checking for document.cookie would have to be done on the server-side.

The bottom line is that it is just totally impractical to manually approve every script that contains document.cookie. What happens when someone uploads a new version? Do we have to wait for that to be approved before it is listed again? Let's not turn Userscripts.org into Addons.mozilla.org

1 point
login to vote
Posted Apr 13, 2009

--Under the old system there were 0 cookie grabbing scripts for keyword "neopets".

How many scripts were there for neopets?

There were 0 cookie grabbing scripts out of around 90.
As of right now, there are 18 out of 123. A little less than half of the scripts since the unlisted feature was removed have been cookie grabbers.

Hell 3 out of my 16 scripts use it.

You should try using GM_setValue and GM_getValue instead. If you're worried about making the scripts work in browsers besides Firefox, greasekit emulates GM functions and there is Opera Userscript for Opera users.

Highlighting is done on the client-side. Checking for document.cookie would have to be done on the server-side.

Of course. If that can be written to work client side, think of all you can do server side.

As far as manually approving unlisted scripts... whatever happened to the idea of having a real community Reviews Discussion. There's a help file for unlisted scripts unlisted scripts. It says there to email admin. There are a lot of people that care about us.o and are willing to help with the workload.

There are currently 405 pages of harmful/spam scripts. How many will still be there next week?

1 point
login to vote
Jesse Andrews article's author
Posted Apr 15, 2009

@nungryscpro,

you are correct that something needs to be done about this. And thank you sizzlemctwizzle for explaining that this is a community site.

I've been slow to expose the controls needed, since I'm afraid of getting the balance wrong... Instead of worrying about getting it perfect on the first shot I need to release a first cut so that improvements can be made.

Thank you both (and others before) for caring enough to help improve this situation!

1 point
login to vote
Anagoge User
Posted Apr 17, 2009

Please consider deleting/editing the basically useless script here:
http://userscripts.org/scripts/show/46036
It is basically spam because it shows up for every www.* site in Greasefire and makes greasefire much less useful. The review and discussion there also state this case. Thanks!

1 point
login to vote
Posted Apr 21, 2009

What about implementing a function in this site that enables to user to vote whether a script is malicious and to show how many users voted the script as malicious (and how many as not malicious) and to show this (somehow highlighted) just below the "Install"-link?
This would help to prevent users from downloading propably malicious scripts.
Maybe there could also be a message-box that asks you if you really want to install a script if there were enough votes (f.e. more than 20) and more than half of the votes say the script is malicious.

2 points
login to vote
Jesse Andrews article's author
Posted Apr 22, 2009

n-regen,

We are working on that right now. If you look there is now a "issues" tab that lets you vote and comment on a script's badness.

1 point
login to vote
Posted May 2, 2009

Great new page updates, tags and layout improvements! Keep up the good work [it's much appreciated!]