NPM New UI - This package has no keywords, version 0.0.0


Looks like every item in the npm Package Manager dialog has (This package has no keywords.) line.

Is that statement accurate or is the parsing of keywords not working correctly?

If it's parsing correctly, and these packages really don't have any keywords, then I think we should get rid of that line. It's not valuable enough (I couldn't find a package that has keywords) and makes each entry in the list taller than necessary.

Edit: It may be a problem with parsing, because they all show version 0.0.0


file attachments

Closed Mar 18, 2014 at 8:48 PM by huguesv


BartRead wrote Feb 27, 2014 at 7:33 PM

I can't repro this. Was it only one version of VS or both 2012 and 2013? Also, had you just refreshed the catalog? I.e., was this happening before your most recent refresh, or only after you refreshed?

If I had to guess I'd say the npm repo was returning junk in response to the search query. We could do something about this but it's not a particularly quick fix. The easiest thing may be to allow the user to revert to the previous version of the package catalog. Of course, eventually, the repo will start behaving again and a refresh will fix the problem.

BartRead wrote Feb 27, 2014 at 7:35 PM

Oh, I should say that the norm is that packages DO have keywords so the no keywords line should be relatively rare. I include the keywords in the display since they're one of the fields the search is applied to: if they're not there it may be unclear to users why some results appear (search term may not match module name or description, but only keywords).

huguesv wrote Feb 27, 2014 at 9:13 PM

That's how it looked before and after a refresh. I just got the same result on a different machine. It may have been broken with the json assembly version reference change that was made yesterday that's not on codeplex yet.

If it was a parsing error that causes the 0.0.0 and no keywords, it would be nice to alert the user that there was an error.

BartRead wrote Feb 28, 2014 at 3:38 AM

I don't think it's a parsing error. I've pulled in all the latest changes (not the JSON one, obviously) and tried several refreshes and it seems fine. The refresh actually failed once (and gave an error/reverted to the old results, as expected) but I think the repo's been flaky for the past 12 hours or so as I had issues installing packages at random earlier.

BartRead wrote Feb 28, 2014 at 3:39 AM

Btw, I have a test for the parsing in the NpmTests assembly that parses out information for around 48,000 packages so if there were a problem with the parsing I'd expect that to catch it.

BartRead wrote Mar 4, 2014 at 6:39 PM

Nope, you're absolutely right (my apologies): it IS a parsing problem.

It's happening because the output from npm search has changed format between Node 0.10.22 (not sure which npm version that corresponded to) and Node 0.10.26, which ships with npm 1.4.3. I've been trying to track down what happened and when to see if there's anything else I need to take into account before I fix this but, thus far, unsuccessfully.

The npm is VERY out of date: https://www.npmjs.org/doc/changelog.html

Unfortunately the published version of the docs at https://www.npmjs.org/doc/cli/npm-search.html is obviously much newer than the version in the repo at https://github.com/npm/npm/blob/f0756e50236fdf6bb17f2d7fe31573565927a7a2/doc/cli/npm-search.md so I can't see when this changed. GitHub search hasn't proven to be much help trawling through the code either.

Anyway, what's changed?
  • DESCRIPTION and AUTHOR fields are now trimmed, unless the --long flag is used, in which case they're split across multiple lines.
  • DATE field no longer includes the time of day
  • There's a LOT of extra whitespace in the output - all the lines are now exactly 1018 characters long
  • This means that the cache file is now 8x as big as it used to be! (63MB as opposed to 7MB)
  • It also means that including the --long flag doesn't make the file that much bigger because of all the extra whitespace! (86MB)
(I can't help thinking the last two items are the result of a bug - there's really no sensible reason for all that extra whitespace. Fortunately I think compression on the web request means it doesn't take disproportionately longer to return the information. Moreover, if it weren't for all the extra whitespace, the --long flag would be completely unnecessary since the overall size of the output would be much lower.)

I've added two new test data files in https://nodejstools.codeplex.com/SourceControl/network/forks/BartRead/rgnpm01/changeset/06e8edfa62f75540c2bbd6014f8fd8d62a2370bd.

One is for the standard npm search output; the other for the output with the use of the --long flag. I don't think we should use the --long flag yet because not everyone will have a version of npm that supports it. I'm assume the self signed certificate error will have forced a lot of people to upgrade, but people who weren't using that certificate won't need to... which means their npm search output will be in the same format we can already handle.

This means I need to support both the old output format and the new (short) output format.

BartRead wrote Mar 4, 2014 at 6:46 PM

UPDATE: the right place to find the docs is https://github.com/npm/npm-www, however there's still no indication of when the changes actually happened.

BartRead wrote Mar 4, 2014 at 7:00 PM

I've filed an issue about the padding in npm https://github.com/npm/npm/issues/4835 but we'll still have to handle it correctly in NTVS as people are bound to be running with this version for a while.

BartRead wrote Mar 4, 2014 at 11:48 PM

Should all be fixed as of https://nodejstools.codeplex.com/SourceControl/network/forks/BartRead/rgnpm01/changeset/307acd79c4730dc5fc5d3a2d94aab2c125443916.

I've modified the parsing to handle npm search output both pre 1.4.3 and from 1.4.3 onwards. I've also added more tests for 1.4.3 and later, to deal with a lot of extra corner cases and weirdness (names split across lines, missing fields, truncated versions, etc.), and split up the existing tests more to provide finer grained coverage of the parsing.

Unfortunately we'll continue to be sensitive to changes in this output format so I'd suggest more tests will need to be added as new versions of npm are released.

This is a PITA because I'm not aware of any other way to get a complete list of packages at present. Still, for now, it's fixed. Resolving this bug.

dinov wrote Mar 7, 2014 at 8:05 PM

Fixed in changeset 9398932c479707b7d09504448444b3e388593c4c

dinov wrote Mar 7, 2014 at 8:05 PM

Fixed in changeset c22ff68ba2dcf2e0a2ffaa26be00ba238ac3f4bf

dinov wrote Mar 7, 2014 at 8:05 PM

Fixed in changeset 307acd79c4730dc5fc5d3a2d94aab2c125443916

dinov wrote Mar 7, 2014 at 8:05 PM

Fixed in changeset ef74d1a0ff0b675d48d683102c4b9722f1dc357f