Python to extract the @user and url link in twitter text data with regex

Question

There is a list string twitter text data, for example, the following data (actually, there is a large number of text,not just these data), I want to extract the all the user name after @ and url link in the twitter text, for example: galaxy5univ and url link.

   tweet_text = ['@galaxy5univ I like you',
    'RT @BestOfGalaxies: Let's sit under the stars ...',
    '@jonghyun__bot .........((thanks)',
    'RT @yosizo: thanks.ddddd <https://yahoo.com>',
    'RT @LDH_3_yui: #fam, ccccc https://msn.news.com']

my code:

import re
pu = re.compile(r'http\S+')
pn = re.compile(r'@(\S+)')
for row in twitter_text:
   text = pu.findall(row)
   name = (pn.findall(row))
   print("url: ", text)
   print("name: ", name)

Through testing the code in a large number of twitter data, I have got that my two patterns for url and name both are wrong(although in a few twitter text data is right). Do you guys have some documents or link about extract name and url from twitter text in the case of large twitter data.

If you have advices about extracting name and url from twitter data, please tell me, thanks!

Answer 1

Note that your pn = re.compile(r'@(\S+)') regex will capture any 1+ non-whitespace characters after @.

To exclude matching :, you need to convert the shorthand \S class to [^\s] negated character class equivalent, and add : to it:

pn = re.compile(r'@([^\s:]+)')

Now, it will stop capturing non-whitespace symbols before the first :. See the regex demo.

If you need to capture until the last :, you can just add : after the capturing group: pn = re.compile(r'@(\S+):').

As for a URL matching regex, there are many on the Web, just choose the one that works best for you.

Here is an example code:

import re
p = re.compile(r'@([^\s:]+)')
test_str = "@galaxy5univ I like you\nRT @BestOfGalaxies: Let's sit under the stars ...\n@jonghyun__bot .........((thanks)\nRT @yosizo: thanks.ddddd <https://y...content-available-to-author-only...o.com>\nRT @LDH_3_yui: #fam, ccccc https://m...content-available-to-author-only...s.com"
print(p.findall(test_str)) 
p2 = re.compile(r'(?:http|ftp|https)://(?:[\w_-]+(?:(?:\.[\w_-]+)+))(?:[\w.,@?^=%&:/~+#-]*[\w@?^=%&/~+#-])?')
print(p2.findall(test_str))
# => ['galaxy5univ', 'BestOfGalaxies', 'jonghyun__bot', 'yosizo', 'LDH_3_yui']
# => ['https://yahoo.com', 'https://msn.news.com']

Answer 2

If the usernames doesn't contain special chars, you can use:

@([\w]+)

See Live demo